Appliance update: texisScripts-25.0.2, applianceman-25.0.2, texis-8.01.1667348496, thunderstonePatch-2.20

Post Reply
User avatar
Kai
Site Admin
Posts: 1271
Joined: Tue Apr 25, 2000 1:27 pm

Appliance update: texisScripts-25.0.2, applianceman-25.0.2, texis-8.01.1667348496, thunderstonePatch-2.20

Post by Kai »

A new set of appliance updates are available: texisScripts-25.0.2, applianceman-25.0.2, texis-8.01.1667348496, thunderstonePatch-2.20.

texisScripts-25.02 provides these features and fixes:
  • Admin accounts are now always listed in alphabetical order
  • Creating a profile can give getDataspaceDir ... FOP_EDOMAIN error if a /data* mounted filesystem is smaller than 1GB total, and profile creation may prefer smaller such disks for dataspace too much; fixed
  • Exponential notation could appear for some floating-point numbers greater than 10^6 (e.g. populating a Parametric Number field, Maximum Refresh Time); now only used if value is greater than 10^12, and 12 digits of precision instead of 6 are used
  • Fixed DFS file server walks failing to start up in primer system
  • Your IP: in System Information's Client section now properly take X-Forwarded-For into account
  • <meta property> and <meta itemprop> tags now available in walks (e.g. for Data from Field)
  • Last-Modified and Expires now obtained from <meta property> and <meta itemprop> if header, <meta name>, nor <meta http-equiv> available. Same for Keywords and Description, which additionally come from <meta property="og:..."> if not found elsewhere. Test Fetch now shows <meta property> and <meta itemprop> values.
  • Fixed Search Timeout setting not affecting metasearch frontend's fetch timeout to backend targets
  • Support Commands now receive additional context from the running system
  • Modifying best bets on List/Edit URLs Details pages now uses the modern, undoable interface
  • Fixed Test Fetch of http-post:// URLs not performing the actual POST http transfer
  • Fixed replication when restoring a backup of profiles that already exist
  • Best Bets are now included in Settings replication
  • Removed ability to rename bestbet groups, necessary for best bet replication
  • Added ability to create ACLs for Static Content and Client Certificates
  • Added ability to create ACLs on admin-created directories inside Static Content
  • Added highlighting to every other row in Profiles list
  • Diskspace Viewer and Cleanup now properly support appliances with multiple dataspaces
  • Fixed Test Fetch not applying the same link sanitization that walks would apply
  • Test Fetch option Fixed font added, to print all values in fixed-font and preserving spaces. Wrap long lines now also does so in large values like Text
  • Added Keep Selectors, Ignore Selectors to keep or ignore HTML elements' text via CSS selectors; defaults to ignoring <nav>, <header>, <footer> elements. Keep Tags renamed to Keep HTML Strings to clarify what it searches for (simple text strings).
  • Keep HTML (Alt Text, <STRIKE>, <DEL>, <FORM>) settings removed; can now be accomplished with Ignore Selectors. Existing profiles with non-default values for these settings will have equivalent Ignore Selectors values added.
  • File crawls now get File-Attributes header with Windows file attribute tokens
  • Added best bets support to dataload API
  • Send Profile Settings in Replication Tools now sends Best Bets along with profile settings
applianceman-25.0.2 updates the documentation.

texis-8.01.1667348496 provides these features:
  • Fetches/crawls now use OpenSSL 3.0.7; fixes CVE-2022-1292, CVE-2022-1343, CVE-2022-1434, CVE-2022-1473 (these do not affect appliances' use of OpenSSL 3.x). Also fixes CVE-2022-3602 and CVE-2022-3786: neither affect appliances: while texis RPM packages from 2021-10-11 until this update use OpenSSL 3.x < 3.0.7 (affected by the CVEs), it is only used for crawls, and no remote certificates are validated during crawls.
  • File types plugin (anytotx) now shows <meta http-equiv,property,itemprop> values (in addition to <meta name>) from HTML documents
  • Formatted text from HTML containing <figure>, <svg>, <canvas>, <article>, <aside>, <figcaption>, <footer>, <header>, <main>, <marquee>, <nav>, <section>, <details>, <summary>, <command>, <ruby>, <audio>, <video>, <template>, <slot>, <spacer> tags may change whitespace, e.g. space to newline(s); re-walked unchanged pages may thus change hash and appear to be changed
  • Some HTML formatter changes involving priority of <div>/<span>/<table>/<tr>/<td>/<th>/<tfoot>/<thead> (and some others like <optgroup>); could possibly affect whitespace in formatted text and thus change crawled pages' hashes
  • <img alt="..."> text, and widget whitespace (e.g. double newline for <p>), were still being printed when they should be suppressed (e.g. inside <style>...</style>, ignoretextselectors/Ignore Elements matches, etc.); fixed. Single space is still printed, to maintain separation of potential bracketing non-suppressed words. Caveat: fetched pages' content (including whitespace) may change, and thus hashes when walking.
  • Upgraded to exiftool version 12.47: gets JPEG comments for anytotx plugin
thunderstonePatch-2.20 provides some system updates, and is needed by texis-8.... RPMs.

Note: Some older updates (listed below) were released late; their functionality is also part of (or superceded by) the above updates. These include:

texis-8.01.1651095857, which provides the following (part of or superceded by the above texis update):
  • Updated fetch lib to OpenSSL 1.1.1i to fix CVE-2020-1971
  • Updated unrar to 6.0.3 for fixes
  • ExifTool upgraded to 12.30
  • Updated fetch lib (for crawls) to OpenSSL 1.1.1l to fix CVE-2021-3711 and CVE-2021-3712, neither of which affect Texis nor crawls
  • Some robustness improvements to DNS lookups, especially under Windows
  • With Best Bets enabled, a user query containing one or more consecutive dashes or plus signs with space or begin/end of query to both sides, might cause an ABEND; fixed. Workaround is to disable Best Bets, or pre-process the query to avoid such queries.
  • Crawls/fetches now use OpenSSL 3.0.2; fixes CVE-2021-4044 (might cause SSL connects to fail; affects Texis versions from 2021-10-11 through this fix) and CVE-2022-0778 (possible infinite loop during cert checks; should not affect Webinator/Appliance generally as client certs not checked during crawls)
  • Texis version 8.01 released:
    • IPv6 crawling support added (with texisScripts 24.0.1 or later)
    • Longer admin passwords supported, more secure hashing (on password create/change after this update). Reboot recommended (after stopping all walks) to ensure password table is updated to enable this feature.
  • Under Linux, taskmonitor process can get stuck waiting for a child process that appears to be (from ps cmdline args) a copy of itself; fixed
  • Fetching a page that redirects may cause an ABEND; fixed. Affects versions from 2020-09-28 and later, prior to this fix.
Note: During application of this update, the GUI might refresh to a Script not found or Vortex error page. If this happens, wait 30 seconds and re-load the page. The error should be transient and benign.

texisScripts-24.1.0, which provides the following (part of or superceded by the above texisScripts update):
  • Fixed viewing performance log showing display name for token [dataFromField] not found
  • Fixed URL deletions with Ignore Case=Y not matching in some scenarios
  • Improved walk efficiency when indexing multiple servers in the same profile
  • Fixed restoring backups of existing profiles with per-profile replication settings sometimes using wrong replication settings
  • Fixed race condition where a replicated Finalize Walk that fails during initialization could have no pid set in its task list entry
  • Fixed rare race condition on Walk Status when the status page starts rendering before a walk starts up, and the walk finishes before Walk Status finishes rendering
  • Fixed replication of refreshed URLs that weren't modified not replicating their meta data correctly
  • Clicking on a URL in List/Edit URLs results might have brought up an empty-URL item instead of the desired-URL item, if an empty-URL item existed (e.g. due to Dataload with empty URL); fixed
  • Some long-running daemons (task monitor, Results Cache manager) now exit sooner when commanded to or change detected
  • Now defaults to Off-Site Components=N in newly created profiles
  • Fixed Show Authorization Info not clearly showing cached Authorization URL checks
  • Fixed search XML output displaying an erroneous through of X summary box when displaying past the end of results
  • Fixed restoring backups with profiles and thesauruses not getting assigned to the profiles properly
  • Fixed replication sender possibly exiting just as items are added to the replication queue
  • Improved behavior when search user is provided malform redirect URL that attempts to execute JavaScript
  • No longer allows admins to turn off HTTPS server while using HTTPS to access the admin interface
  • Login Verification URL can now be used with Forward Login Cookies Results Authorization type
  • Deleting a profile now properly stops all tasks running in the profile first
  • Admin interface will now check if custom search XSL Stylesheets are well formed, and will not save them if there are errors
  • No longer misleadingly allows modifying Parametric Fields on metasearch profiles
  • Requesting an admin page when not logged in will now redirect to that page after logging in when using HTTPS
  • Detection of missing internal data version is now properly fixed
  • Fixed incorrect merging of spelling suggestions in metasearch
  • Improve reliability of automatic software updates
  • Enhanced refresh algorithm to allow attempting duplicate URLs less often
  • Admin Banner System-Wide Setting was not being shown on some admin pages
  • Enhanced metasearch merging of spelling suggestions to provide more accurate suggestions
  • Fixed Keep/Ignore Tags causing some unusual pages to leave html tags in their content
  • Fixed archive log cleanup somtimes removing newer log files before older log files
  • Updated JQueryUI to 1.13.1
  • Fixed javascript dropin using customer site's jQuery with jQuery.noConflict() logging the error '$ is not a function'
  • Version 24.0.1: Support for IPv6 crawling added (in conjunction with texis 8.01 or later RPM)
  • Changing admin password was failing to change Webmin password and was giving exec command sudo returned exit code 1 error; fixed. Only affects scripts version 24.0.1 and earlier in combination with Texis version 8, so no appliances should be affected.
Note: All of these updates are currently only available for Gen3 (2013) and later appliances, pending a thunderstonePatch update for older appliances.

Updates can be found via the System -> System Setup -> Update Software menu item.
Post Reply