ENCP release notes, from v3_3 to v3_4
Encp changes:
=============
Added the switch --check. When this option is used, encp goes through all
of the pre-transfer setup, but does not actually perform the transfer. If an
error is found, it is reported like a normal encp error. If the user is
returned success, it does not guarantee that re-running the encp without
--check will also be without errors.
A number of old metadata 'formats' were discovered that encp was not
able to handle. They are now accepted. One format, was that all bfid
brands by convention should be 4 bytes but some were found to be 3
bytes long. The other was that some bfids have an "L" appended to them.
Transfer failures are now sent to the accounting database along with
the successes.
New error, handle CRC_DCACHE_MISMATCH, occurs when the crc the dCache
puts into layer 2 does not match the CRC calculated while attempting to
write the file into Enstore. This is a serious error. This error is
known to occur when a user keeps rewriting over-and-over again the same
file into dCache.
New error, UNCAUGHT_EXCEPTION, occurs when previous versions of encp
would have given a traceback to stdout. Now when such an error occurs
the traceback dump is instead sent to the log server, zero length files
are cleaned up and the accounting server is notified of the failure.
NOTE: This does not work for "kill -9".
Misc.:
======
A threading bug was discovered that affects all non-Linux systems and newer
Linux systems with NPTL. Older Linux systems (with Pthreads) would return
a different process id (PID) for each thread in the process. This behavior
does not conform to the POSIX standard. Enstore/encp would use this value
as a unique key for keeping messages to and from different threads separate.
This does not work on POSIX compliant systems because all threads are
returned the same value. Now, a unique thread number is used instead of the
process id. Almost everything in the enstore and encp commands is single
threaded and did not trip over this issue.
A bug was discovered that was allowing the enstore servers to send replies
to offsite computers despite the department policy to not allow this. The
production servers have been patched for some time, but this is the first
cut encp product that also enforces this policy.
The ecrc program has some new switches:
-0: Use the enstore 0 seeded adler32 algorithm. (default)
-1: Use the standard 1 seeded adler32 algorithm.
-d: Output crc as decimal number. (default)
-h: Output crc as hexidecimal number.
-v: Output additional information.
The ddencp program has a new switch:
-c: Attempt to remove the file's contents from the buffer cache in memory.
This is useful when testing read/write rates of the disk. Not all
systems support this functionality.
-S: Open regular files with O_SYNC passed to open(2).
-D: Open regular files with D_SYNC passed to open(2).
-R: Open regular files with R_SYNC passed to open(2).
The -v switch has been modified to additionally report how many pages of
the file exist in the file buffer cache in main memory. This only works
on systems that support the mincore(2) function.
Detailed cvs commit logs
========== Trace.py ====================================================================================
Cleaned up a __pychecker__ warning about unused arguments. Also, modified handle_error() in a way that should prevent resource leaks from the local tb variable problem.
========== accounting_client.py ====================================================================================
add storage_group to log_encp_error()
change the interface of log_encp_error() to accomodate multi-interfaces nodes
========== accounting.py ====================================================================================
take care of error being None in log_encp_error()
bug fix
add storage_group to log_encp_error()
add log_encp_error
========== alarm.py ====================================================================================
fix another warning
remove warnings
========== alarm_client.py ====================================================================================
Disabled a pychecker warning with __pychecker__.
Fixed the default server_name to be MY_SERVER instead of None.
========== alarm_server.py ====================================================================================
Modified the servers to utilize the new configurable hostaddr (in)valid ip list functionality.
========== callback.py ====================================================================================
Handle EINTR differently if returned from select(). Also fix a math bug if the "select duration" error occurs.
Added diagnostics to help understand various network errors we are seeing.
========== cern_wrapper.py ====================================================================================
fixed a bug causing exception in volume_labels for labels with lengt more than 6 chars. For instance PRO539L1
========== checkdb.py ====================================================================================
take out createuser which was a mistake
add user root
========== cleanUDP.py ====================================================================================
Added some comments. Cleaned up lines longer than 80 characters. Added the linux 2.4 kernel to the non-linux case since it seems that the non-standard behavior linux used to do was fixed.
========== configuration_client.py ====================================================================================
Fix a backward compatibility issue. Old clients can now use the new server and will receive the old reply ticket format.
Use the hostaddr (in)valid domain functionality.
fix help output of enstore config
remove ticket parameter for get_library_managers
Added the --timestamp function for a user to query when the current configuration was loaded. This information is now used to determine if cached config information is still valid.
========== configuration_server.py ====================================================================================
Fix a backward compatibility issue. Old clients can now use the new server and will receive the old reply ticket format.
Modified the dump() and lookup() functions to support the hostaddr (in)valid domains functionality.
Added support for the configuration server to remember when it last loaded the configuration file. This information can be queried.
========== delete_at_exit.py ====================================================================================
Only instantiate the csc when the _deletion_list_bfids is non-empty.
Just some cleanup and comments.
========== drivestat_server.py ====================================================================================
Modified the servers to utilize the new configurable hostaddr (in)valid ip list functionality.
========== e_errors.py ====================================================================================
Added the UNCAUGHT_EXCEPTION error.
Added CRC_DCACHE_MISMATCH to the list of errors that can be reported from encp. This differentiates this type of error from CRC_ENCP_MISMATCH which was used for both situations.
========== encp.py ====================================================================================
Comment out the --copy option for the v3_4 cut.
bumping version to v3_4 because of encpCut
Added the --copy switch for encp. This switch allows for encp to write N copies of the file to different tapes.
Fixed an error reported. Two cases exist. One is when get_file_clerk_info() is passed a ticket and not just the bfid when an error occurs. The other is when get_volume_clerk_info() also has an error when a ticket (and not just the external label) is passed to it. When the error string was put together the entire ticket was used instead of just the bfid or volume.
Fixed e_access() to handle or-ed os.R_OK, os.R_OK and os.X_OK values. Not just one of them at a time.
Use the new code in enstore_functions2.py 1.12 to handle reading filenames like: stkensrv2:7500//pnfs/test/testfile. Also, fixed a bug in reading some bfids. Some 2000 bfids from D0 have "DMS" bfids instead of "D0MS". Encp had problems with these bfids. The brand and bfid handling code has been fixed to handle brands with lengths other than that of four bytes.
Modified print_data_access_layer_format() to check for sub-tickets to actually be dictionaries. There seems to be a way for a string to be named one of these sub-tickets that causes tracebacks.
Use the new accounting server ability to handle None in the message field for log_encp_error().
Modify how encp, ensync and enmv use the option.Interface class's print_usage() function. All command line parsing errors, should now give an error return code.
Don't pass None to log_encp_error() for the error paramter. Pass "" instead.
Fixed the cause of an UNCAUGHT_EXCEPTION. The if...else branch did create a situation where an uninitialized variable could be used.
Bug fix in the way encp handles bogus libraries. If the library specified does not exist, an UNGAUGHT_EXCEPTION would occur. This is now handled.
Fixed a handle_retries bug such that the queue size is now correctly determined on errors. Also, when such error occurs, it is not deleted from the queue since the 'completion_status' ticket field is now used.
It might be possible that an error while reading could be forgetten about when the exit_status is printed/returned. Modified the if...elif...else statement to protect against wrongly falling into the 'else' section.
Encp resends on writes were being done twice instead of just once. This has been fixed. Also, the messages to the user and log file specifing that encp is trying to submit have been "cleaned up" and only occur from one spot in the code now, namely submit_one_request(), instead of all over.
Added "get" option --skip-deleted-files. This option will cause get to skip over deleted files.
Bug fix. Now it handles exceptions raised when the file has been deleted before it attempts to clobber layer 1 & 4 during a dcache write.
Fixed the message string that is created when a volume is NOACCESS (the volume name was listed as the entire work_ticket as a string). Also, modified where print_data_access_layer_format() looks for the volume in the ticket to print the most accurate information. It was not handing the case where only the 'external_label' in the VC sub-ticket was known, but none of the others.
The switch --bypass-filesystem-max-filesize-check was broken. Fixed it.
Added support for encp to record failures in the accounting database. Added the UNCAUGHT_EXCEPTION error to give to users when encp would otherwise traceback.
Fix a "get" bug. When get would attempt to read a file with only a partial entry in the file database and no corresponding pnfs file; the new pnfs file would not be created because the pnfs class instance wasn't getting instantiated correctly. Added a pnfs.Pnfs() call and bug fixed.
Handle an AttributeError exception for "get" when the pnfs file does not exist when entering the check_crc() function. This check is for the dcache layer 2 check.
Fixed use of the --check option. I had become broken.
Fixed two bugs relating to ignoring of the --library and --file-family-wrapper switches. The first was only cosmetic. The First log file message printed the wrong values. The second was due to always looking at the output directory tags when checking the maximum size of the library and file family wrapper.
Modified the control socket error detection section in handle_retries(). It might be eating messages. Now just check if the queue is empty or not after the select().
Modified encp to determine the next transfer similar to how "get" does so. This involves using the 'completion_status' field in the request ticket. This work is done it preperation of having an API (for automigration).
Added CRC_DCACHE_MISMATCH to the list of errors that can be reported from encp. This differentiates this type of error from CRC_ENCP_MISMATCH which was used for both situations.
Modified when/how the control socket is accessed before transfer_file(). There might have been eaten final dialogs for really small files. This entailed two new functions: stall_read_transfer() and stall_write_transfer() that encapsulate mostly existing inline functionality.
Added code to detect (with multiple file transfers) if two input files will be copied to the same output file. (i.e. same name; two input dirs)
I changed the timeout values when checking the control socket for any unexpected problems. Instead of 5 seconds it is now 1 second.
Misspelled max_resumit as max_resubmits.
Removed a comma that mylint.py did not like.
Removed dependance on brands to determine if a file clerk knows about a bfid. Made a change to use the newer configuration_client.py caching mechanism. Started adding accounting_client code to have encp record information about failure to the accounting database. Still commented out. Moved max_attempts() to inside create_read/write_requests(). The retry/resubmit (and maxes) information is now placed inside a sub-ticket. Moved the external_label checks around. This is to avoid a check after each file (which is painful for get). Bug fix for get; input filenames were always getting the sequetial filenames even when the real names were given via --list.
get_csc() can raise exceptions. Some places in encp/get did not catch them. They are now caught.
Handle bfids with "L" appended to them correctly. This would cause an exception and the exception handling code had another bug that caused a traceback. Also, handle files where the file database metadata has None for the crc. This was also causing tracebacks. Lastly, make a preparatory change to EncpError to place the "status" error inside the optional ticket if not already done so.
Fixed two bugs regarding encp not recognizing some bfids. Some older bfids (Nov 2000 were when the sample file's bfids were created) appended a long() "L" to the end of the bfid. This L is consistant between the pnfs metadata and the file database. Thus, encp will now recognize these correctly. The other bug was in the error handling code that tried to use the wrong undefined variable when forming the error string for the first bug mentioned.
Fixed a bug where Pnfs.__init__() would succed, but if get_path() was passed as the parameter the empty string, a traceback would occur. Pnfs.get_path() can now throw ValueErrors (in addition to OSErrors or IOErrors), which are now handled by encp.
Modified the code to deal with some occasional errors. These occur when __get_fcc() (and __get_vcc()) return None for the volume/bfid information. This can occur (correctly) for some cases; like None was passed into the function for the volume/bfid parameter. However, for these error cases there should have been a valid value. These changes should handle the error and simultaneously determine what is actually happening.
========== enmv.py ====================================================================================
Modify how encp, ensync and enmv use the option.Interface class's print_usage() function. All command line parsing errors, should now give an error return code.
========== enstore_erc_functions.py ====================================================================================
remove unused param in read_erc method
========== enstore_files.py ====================================================================================
generation of per group tape ingest rates html poage
change NONE to ENONE
========== enstore_functions2.py ====================================================================================
Provide underlying support for encp to handle reading filenames like stkensrv2:7500//pnfs/test/testfile. This was done by adding a function fullpath2(). Also, some common code between fullpath() and fullpath2() was put into expand_path().
Catch a socket.error exception. This function has the ability to fail from "to many files open", though I'm sure there are others errors too.
Cleaned up the code w.r.t pychecker warnings. Unused imports, functions that do nothing, unsued variables, etc.
========== enstore_html.py ====================================================================================
generation of per group tape ingest rates html poage
change NONE to ENONE
========== enstore_overall_status.py ====================================================================================
fix warnings
fix NONE
========== enstore_show_inv_summary_cgi.py ====================================================================================
update special files for each system
add QUOTA_ALERT
add RECYCLABLE_VOLUMES
add MIGRATION_STATUS
add MIGRATED_VOLUMES
========== enstore_show_inventory_cgi.py ====================================================================================
update special files for each system
add QUOTA_ALERT
add RECYCLABLE_VOLUMES
add MIGRATED_VOLUMES
========== enstore_start.py ====================================================================================
fixed redirection of the output. Estart did not work without it
remove ticket parameter for get_library_managers
========== enstore_up_down.py ====================================================================================
add barebones ticket generation when log/alarm/config server down and ticket needed
remove pdb
fix bug where servers were not being 'seen down'
========== ensync.py ====================================================================================
Modify how encp, ensync and enmv use the option.Interface class's print_usage() function. All command line parsing errors, should now give an error return code.
========== entv.py ====================================================================================
Added code to re-exec() the entv process when the memory leak grows to big.
Fixed the popup window of mover information when a user clicks on the mover state. Before it would traceback.
Added some retry capability to mover status requests. After 5 seconds another request is sent out. After 5 retries we give up.
Recent fixes made entv work faster, they also introduced sizable memory leaks. This patch should fix the leaks without adding slowdown. 1) Reworked the reinitialize()/window_killed() relationship. The most important thing is that destroy() function gets called. 2) Functionalized the garbage collection and reporting. 3) Added a new garbage collection warning. 4) Fixed a bug if all system names on the command line are invalid. 5) Eliminated the request_mover_status() thread. All communication is now done through handle_messages(). 6) Part of the --generate-messages-file functionality is still broken. This part is the clean ending of the output.
Fix a newly introduced bug that prevents the display from working after a reinitialization.
All network communication is handled in handle_messages(). This simplifies the sychronization between this thread and the main thread. Movers in offline state are now displayed correctly. Same for those in Unknown state. Recent optimizations caused this.
Attempting to eliminate stubborn resource leaks.
An attempt to plug another resource leak.
Fixed the bug where entv moves upward on the screen when it reinitializes. The update() function needed to be called for the root window object, not just the Canvas object. Hopefully, fixed the IRIX issue where the choosen font size is way to big. The font family requested was always arial. The problem is that few UNIX systems have arial fonts on them. Linux seemed to pick a good substitute but IRIX did/does not. Now uses Helvetica font family. Added a startup_lock. This should allow for a faster startup. This thread prevents the "netowrk" threads in entv.py from doing anything until the main thread is in mainloop(). When multiple enstore systems are specified, it also keeps the resources down by letting only one go at a time. There was a sizable resouce leak. The csc's that were created in the csc_command() function where not being reclaimed. Despite what gc.collect() was saying. These cscs are now explicitly destroyed in destroy() to fix the problem. A ton of speed improvements have been added. Too many to name. Most are of types described in Chapter 17 of "Python and Tkinter Programming." A lot of profiling was done to aid in finding the slowest sections.
Fixed two bugs. First, is the bug where entv on re-initializations will move upward on the screen the width of the titlebar. This is due to an inconsistancy in which object the geometry values are taken. The second is to add a starup lock that prevents the mainloop from asking movers about their state if the mover_status thread is still running. This greatly improves startup time as the different threads are not duplicating work.
remove ticket parameter for get_library_managers
Added an option to entv: --capture-timeout This option allows for entv to specify how long (in seconds) --generate-messages-file should continue to display new transfers.
Added the --generate-messages-file option to entv. This option allows for the user to have entv output to standard output the messages it used to draw the display. This output, if redirected to a file, can using --messages-file (renamed from --commands-file) recreate the earier entv displayed output. Also, fixed the --messages-file (former --commands-file) switch to actually work.
Made changes regarding how configuration information has changed. These changes were necessary after changes in configuration_client.py.
Fixed the --don't show option. It became broken when the config caching functionality was added.
========== event_relay_client.py ====================================================================================
Suppress pychecker warnings. These have been confirmed to be false positives and the code is correct.
========== event_relay_messages.py ====================================================================================
Suppress pychecker warnings. These have been confirmed to be false positives and the code is correct.
remove warnings
========== file_clerk.py ====================================================================================
add show_state()
minor change
new_bit_file() now accepts predefined bfid
same upgradei as info_server/client: shift formatting load to client
Modified the servers to utilize the new configurable hostaddr (in)valid ip list functionality.
========== file_clerk_client.py ====================================================================================
add show_state()
fix tape_list
same upgradei as info_server/client: shift formatting load to client
========== ftt_driver.py ====================================================================================
return more from flush
========== get.py ====================================================================================
bumping version to v1_45 because of sdsscpCut
Fixed an error message. It had incorrect format to value matches for the error string.
Fixed a automounting bug where ENOENT could false be returned. Now a few retries of the os.stat() are performed before giving up with this error.
bumping version to v1_44 because of sdsscpCut
Modified get's use of submit_read_requests() to use 2 parameters not 3. Encp had changed and get is now getting its change. The command line output to terminal and log file was wrong. Due to the way the volume is inserted into the interface class it was being dropped. It now appears in these output lines.
Added "get" option --skip-deleted-files. This option will cause get to skip over deleted files.
Fixed the parameter part of the usage help line. It was suggesting source and destination files not directories.
Added the UNCAUGHT_EXCEPTION error to give to users when get would otherwise traceback.
bumping version to v1_43 because of sdsscpCut
Added a combine_dict() call to make sure the work ticket had the error from done_ticket in it's status field. Otherwise, the error could be dropped. The situation for this was an error occuring in the first place was an error in stall_read_transfer().
bumping version to v1_42 because of sdsscpCut
Cleaned up some unused code any pychcker warnings. (Removed EOD references.)
Modified when/how the control socket is accessed before transfer_file(). There might have been eaten final dialogs for really small files. This entailed two new functions: stall_read_transfer() and stall_write_transfer() that encapsulate mostly existing inline functionality.
bumping version to v1_41 because of sdsscpCut
Moved where external_label checks are performed. For get this was a performance hit. Changed to use the new resend subticket that holds retry/resubmit info.
get_csc() can raise exceptions. Some places in encp/get did not catch them. They are now caught.
Fixed the handling of OSErrors from create_zero_length_local_files(). They were causing a traceback to occur.
========== get_total_bytes_counter.py ====================================================================================
add new libraries
========== hostaddr.py ====================================================================================
Added support for an invalid ip list. Made the (in)valid ip list(s) configurable in the config file. And lastly, cleaned up the __main__() section.
========== info_client.py ====================================================================================
fix tape_list
remove dependency on edb
fixing ls-active
1 shift formatting of the output of get_vols(), get_pvols(), tape_list() and list_active() to client [2] optimize some queries [3] enhance get_vols() with keyed value
info_server.py
========== info_server.py ====================================================================================
allow lower case noaccess, notallowed, deleted in --vols key
minor message change
fixing ls-active
1 shift formatting of the output of get_vols(), get_pvols(), tape_list() and list_active() to client [2] optimize some queries [3] enhance get_vols() with keyed value
Modified the servers to utilize the new configurable hostaddr (in)valid ip list functionality.
========== inquisitor.py ====================================================================================
Modified the servers to utilize the new configurable hostaddr (in)valid ip list functionality.
remove unused param in read_erc method
========== inventory.py ====================================================================================
minor adjustment of print out
add CR in recyclable message
add recyclable list
handle just deleted volumes
add count for migrated volumes
add more information to MIGRATED_VOLUMES
fix a typo
add migrated volumes report
========== library_manager.py ====================================================================================
added processing of lowest priority requests, modified change of state to go to the default state if specified in configuration
Modified the servers to utilize the new configurable hostaddr (in)valid ip list functionality.
fixed indentation problem
fixed rm_active_volume
added total time for active requests
modifications for processing mgration requests
========== log_client.py ====================================================================================
Cleaned up pychecker warnings: unused imports, TRUE vs. True and unused variables. Also, modified lines that were longer than 80 characters to be shorter.
========== log_server.py ====================================================================================
Modified the servers to utilize the new configurable hostaddr (in)valid ip list functionality. The log server now has its own log_func() to log its own log messages.
========== makeplot.py ====================================================================================
Added check to make sure that gnuplot is installed before trying anything.
========== manage_queue.py ====================================================================================
added processing of lowest priority requests
========== media_changer.py ====================================================================================
Modified the servers to utilize the new configurable hostaddr (in)valid ip list functionality.
added logoff command to STK method
========== monitor_server.py ====================================================================================
Turned a print statement into a Trace.log() call. If stdout is closed and the print is executed a traceback would occur.
========== mounts_plot.py ====================================================================================
add library
========== mover.py ====================================================================================
1. Set tape to NOACCESS and offline the mover if it is stuck 2. Offline the mover if get_stats throws an exception.
Added Trace.notify() calls inside assert_vol(). When a volume was asserted entv was not receiving the 'connect' and 'disconnect' messages.
go offline if close_dev failed
fixed processing of dismount on mount of HiPri request failure
Missed an import of event_relay_messages.
Modified the servers to utilize the new configurable hostaddr (in)valid ip list functionality.
do not offline mover if tape thread did non complete for DLT tape
increased delay for dlt drives to synchronize tape and net threads
increased retry TO to 20 s for syncronization between net and tape threads. The problem is seen so far only with DLT tapes
fixed a bug introduced in the last version
Synchronize read_tape and write_client codes. Write_cleint needs to wait until read_tape finishes. The problem was noticed reading DLT tapes. Mover would go into HAVE_BOUND state before tape thread completed, which caused it to go OFFLINE (Tape thread is running in HAVE_BOUND state).
more fixes for race conditions
fix fix race conditions
fix fix race conditions
fix race conditions
it may happen so that for some unknown yet reason the net thread hangs after connection was broken by mover as the consequence the data transfers can not be done until hung net thread is gone. This thread can hang forever to deal with this situation do the following here: check for the presence of the net and tape thread in the IDLE or HAVE_BOUND state. They should not be running. If net thread is running restart the mover. It is a radical measure, but documents do not recommend killing thread with not closed or hung socket. If tape thread is running put mover offline.
Removed the one usage of enstore_functions.is_ok(). It now uses e_errors.is_ok() for the same functionality. This allows for the import of enstore_functions to be removed, since this is_ok() call was the only thing the module was used for.
========== option.py ====================================================================================
Added the --copy switch for encp. This switch allows for encp to write N copies of the file to different tapes.
add show_state()
If print_usage() is passed an error message, return an exit code of 1. Otherwise, on success, return 0.
Added "get" option --skip-deleted-files. This option will cause get to skip over deleted files.
Added an option to entv: --capture-timeout This option allows for entv to specify how long (in seconds) --generate-messages-file should continue to display new transfers.
Added the --generate-messages-file option to entv. This option allows for the user to have entv output to standard output the messages it used to draw the display. This output, if redirected to a file, can using --messages-file (renamed from --commands-file) recreate the earier entv displayed output. Also, fixed the --messages-file (former --commands-file) switch to actually work.
Added --timestamp for config client.
add --full
========== pnfs.py ====================================================================================
Modified the pnfs scan to be faster. In pnfs.py that meant removing access() calls and tweaking the id and parent functons. In scanfiles.py e_access() from encp was copied in and modified a little, uncessesary access calls were removed and the order of pnfs and enstore queries were tweaked.
Added that the parent id is found when the File class is instantiated. This extra information is used in scanfiles.py.
If the correct "p" function is not found, make sure the return code is 1 for error and not zero for success.
Removed a bug in is_pnfsid(). If the current working directory had a filename name the same as its pnfsid (which is true for dcache files) then a false error could have been given.
do not check p_path for consistency
The --tagchown and --tagchmod had the same bug. If the tag specified did not exist a traceback was thrown. The traceback is now handled.
bug fix again
bug fix, again, in set_size()
bug fix again in set_size()
fix a bug in setting the size wrong
take care of size > 2G in class File
Handle user root as a special case in verify existance. Even without other file permissions user root can access a file.
Fixed a bug where Pnfs.__init__() would succed, but if get_path() was passed as the parameter the empty string, a traceback would occur. Pnfs.get_path() can now throw ValueErrors (in addition to OSErrors or IOErrors), which are now handled by encp.
========== quota.py ====================================================================================
handle approaching quota limit
fix --show-by-library
fix --show-by-library
fix --show-by-library
make pychecker happier
show allocated numbers
========== ratekeeper.py ====================================================================================
Modified the servers to utilize the new configurable hostaddr (in)valid ip list functionality.
========== scanfiles.py ====================================================================================
Correctly ignore .bad.* files.
Comment out some profiling code.
Modified the pnfs scan to be faster. In pnfs.py that meant removing access() calls and tweaking the id and parent functons. In scanfiles.py e_access() from encp was copied in and modified a little, uncessesary access calls were removed and the order of pnfs and enstore queries were tweaked.
Commented out the profiling code.
Added a try... except to catch when a directory is deleted. Also, a new function start_check(line) has been added. This is to help encapsulate this common to having the input from a 'file' or from the command line.
Added check to make sure the recored parent id of a file matches the directory that the file is currently in. Also, if a directory itself is OK don't print anything. If an error or warning exists for a directory, still print the message.
Modified the main loop to read from file objects one line at a time instead of doing readlines() and consuming potentially vast resources.
By default will read from stdin if no other input is specified.
Seperated the errors when (1) the file no longer exists and (2) when a directory contains an entry to an invalid/non-existent i-node.
Fix a newly created bug that causes Control-C to be ignored.
Modified the logic to be able to report if both layer 1 and layer 4 are empty, not just the first one found to be empty. Also, modified the error reporting for moved/renamed files; an error is given if layer 4 does not match the file database and a warning if these to values match but differ from the current filepath.
Made files beginning with ".nfs" or ending with "_lock" and error. Skip files beginning with ".bad.". Remove unecessary stat()s from os.path.xxx() calls. Now call os.stat() and check everything from that information. Uses e_access() from encp.py instead of os.access(). (Done as a hack.) Eliminated pychecker warnings.
handle old location cookie format
========== set_lm_noread.py ====================================================================================
Removed two references to enstore_functions.is_ok(). Replaced with e_errors.is_ok().
========== udp_common.py ====================================================================================
Cleaned up pychecker warnings regarding class attributes not being found. The confusion stems from the cleanUDP class acting like a socket without inheriting from the socket object class.
========== volume_assert.py ====================================================================================
Added 'file_family' ticket item to stop the inquisitor from tracebacking.
Fixed errors that resulted in modifiying functions to work with recent encp.py code.
========== volume_clerk.py ====================================================================================
add show_state()
allow lower case noaccess, notallowed, deleted in --vols key
same upgradei as info_server/client: shift formatting load to client
Modified the servers to utilize the new configurable hostaddr (in)valid ip list functionality.
fix a bug that miss counting newly added blanks; page admin when quota is exceeded
making pychecker happy
change wording
1 correct a typo [2] make "quota reached" a separate case [3] show (lib, sg) in quota exceeded message
volume_clerk.py
change alarm message
clean up on alram messages
change alarm message
fix a typo
fix a typo
add advanced alarm when (library, storage_group) approaches its quota limit
fix a typo
fix a bug in using get_sg_counter()
fix a typo
1 handle sg_count for all storage group [2] low water warning and alarm for blanks in common pool
take out paging when a blank is drawn from common pool
add check_state switch for delete_volume()
remove ticket parameter for get_library_managers
fix a typo
let volume clerk to erase files. No contact to file clerk any more
add --full
========== udp_client.py ====================================================================================
One last piece of the udp_client threading problems. This time it is that the __del__ method was using get_tsd(). If the last thread did not use the UDPClient then there is nothing left to destroy. But get_tsd() would create a UDPclient instance just to be destroyed. The fix was to have __del__ check if self.tsd has an element, if so only then delete it (as opposed to create one just to be deleted).
Fixed another threading bug. This time it is when the udp_client object needs to be garbage collected. There was a cyclic reference that prevented the object from being cleaned up.
Fixed a serious bug on POSIX conforming thread implimentations. Older linux OSes would return a different process id when getpid(2) was called in each thread. Conforming implimentations will get the same process id for each thread that calls getpid(2). Using the pid on a conforming OS as a key for a dictionary of thread specific information would lead to all threads dangerously sharing the same socket. Also, in order to test the preceding bug; in the __name__ == "__main__" section the test code was fixed to actually work.
Cleaned up two pychecker warnings. The first was fixed by using the tsd Container class in a slightly different manor. The other was to replace the string UDPError exception with a class UDPError exception.
========== accounting_server.py ====================================================================================
Modified the servers to utilize the new configurable hostaddr (in)valid ip list functionality.
add storage_group to log_encp_error()
add log_encp_error
========== enstore_display.py ====================================================================================
Cleaned up the code that finds the correct library paths.
Expanded use of the display_lock to prevent the possibility of race conditions.
Fixed the popup window of mover information when a user clicks on the mover state. Before it would traceback.
Recent fixes made entv work faster, they also introduced sizable memory leaks. This patch should fix the leaks without adding slowdown. 1) Reworked the reinitialize()/window_killed() relationship. The most important thing is that destroy() function gets called. 2) Functionalized the garbage collection and reporting. 3) Added a new garbage collection warning. 4) Fixed a bug if all system names on the command line are invalid. 5) Eliminated the request_mover_status() thread. All communication is now done through handle_messages(). 6) Part of the --generate-messages-file functionality is still broken. This part is the clean ending of the output.
Fix a newly introduced bug that prevents the display from working after a reinitialization.
All network communication is handled in handle_messages(). This simplifies the sychronization between this thread and the main thread. Movers in offline state are now displayed correctly. Same for those in Unknown state. Recent optimizations caused this.
Fixed a bug when get_up_to_date() does not get something valid from the mover.
Fixed a problem with large file sizes.
The ability for both progress bars became broken in the last update. This fixes it.
Fixed the bug where entv moves upward on the screen when it reinitializes. The update() function needed to be called for the root window object, not just the Canvas object. Hopefully, fixed the IRIX issue where the choosen font size is way to big. The font family requested was always arial. The problem is that few UNIX systems have arial fonts on them. Linux seemed to pick a good substitute but IRIX did/does not. Now uses Helvetica font family. Added a startup_lock. This should allow for a faster startup. This thread prevents the "netowrk" threads in entv.py from doing anything until the main thread is in mainloop(). When multiple enstore systems are specified, it also keeps the resources down by letting only one go at a time. There was a sizable resouce leak. The csc's that were created in the csc_command() function where not being reclaimed. Despite what gc.collect() was saying. These cscs are now explicitly destroyed in destroy() to fix the problem. A ton of speed improvements have been added. Too many to name. Most are of types described in Chapter 17 of "Python and Tkinter Programming." A lot of profiling was done to aid in finding the slowest sections.
Fixed two bugs. First, is the bug where entv on re-initializations will move upward on the screen the width of the titlebar. This is due to an inconsistancy in which object the geometry values are taken. The second is to add a starup lock that prevents the mainloop from asking movers about their state if the mover_status thread is still running. This greatly improves startup time as the different threads are not duplicating work.
Added the --generate-messages-file option to entv. This option allows for the user to have entv output to standard output the messages it used to draw the display. This output, if redirected to a file, can using --messages-file (renamed from --commands-file) recreate the earier entv displayed output. Also, fixed the --messages-file (former --commands-file) switch to actually work.
Handle clearing the percent done display of mover the same way as the buffer part; when undrawing them set the variables holding the quanties to be displayed to None. Thus, when the next transfer is started, None is not equal to 0 and the empty boxes get displayed.
This change sets the current buffer bar size to None when it is undrawn. Without this there is a posibility that on the next file the bar does not get redrawn because it is the same as the remembered size.
Made changes regarding how configuration information has changed. These changes were necessary after changes in configuration_client.py.
========== host_config.py ====================================================================================
Re-implemented the is_route_in_table() function to be faster. To many string.split()s and string.join()s.
Added function clear_cached_routes(). It will reset the _cached_netstat global back to its initial state. Nothing uses it now, but it was useful in cleaning up objects (in the list) for debugging resource leaks.
========== dispatching_worker.py ====================================================================================
Handle differences for the configuration server in the way it works versus the other servers. The difference is that the configuration server does not need notification that there was a new configuration was loaded.
Fixed a security bug. The return value from hostaddr.allow() was being ignored. This would allow dispatching worker servers to reply to requests from places it should not. The value is now honored and such requests are ignored.
========== multiple_interface.py ====================================================================================
Removed some code that resulted in a pychecker warning. Previously, the code would import multiple_interface, which is weird from that module itself. Used the sys.modules[] list to access the modules contents. Also, found another bug in _parse_default(). Fixed that too.
========== generic_server.py ====================================================================================
Handle differences for the configuration server in the way it works versus the other servers. The difference is that the configuration server does not need notification that there was a new configuration was loaded.
Bug fix. Fixed a bug that was causing all server to not cache the contents of the configuration. enable_caching() does not take any parameters, but was being passed one. This caused a traceback caught in a blanket except that did nothing.
remove unused param in read_erc method
========== migrate.py ====================================================================================
change CMS_MIGRATION_DB to 'cms/MIGRATION-9940A-TO-9940B'
cms conforms to standard
handle cms migration_path()
migrate to 'none' if there is no active file
add --bypass-filesystem-max-filesize-check to encp
deal with ghost files
do not check drive any more
allow open target volume to be scanned
1 make sure the file does not have a prefix /pnfs/fs/usr/Migration in final_scan_volume [2] log migration history in the end of each volume migration
delete target file after write failure
Set the last_access of the from volumes to now in the end of final_scan_volume
shorten comment
add --migrated-from and --migrated-to, bug fix
add --migrated-from and --migrated-to, bug fix
add --migrated-from and --migrated-to, bug fix
add --migrated-from and --migrated-to
set comment of 'migrated to' at the end of volume migration
set comment at the end of scan-vol
1 retry once when encountering COPY_TO_TAPE error [2] show location cookie in more log message
print location cookie in final scan volume
check more for volume migration
fix a typo
handle unknown:unknown drives
fix a typo
bug fix
rearrange message coming out fo final_scan_volume()
rearrange message coming out fo final_scan_volume()
take out the debugging code for final_scan_volume()
a few minor bug fixes and a section of debugging code for final_scan_volume()
using poor-man's encp API
log command line
change to final_scan_volume(): 1 check for file family *-MIGRATION [2] check for 'full' [3] move restoring file family to the end of final scan [4] restoring file family only if there is no error [5] returns number of errors
========== volume_clerk_client.py ====================================================================================
forgot to change the tape_list interface
add show_state()
remove dependency on edb
same upgradei as info_server/client: shift formatting load to client
making pychecker happy
making pychecker happy
add check_state switch for delete_volume()
add --full
add --full
========== enstore_functions.py ====================================================================================
Removed the is_ok() and is_timedout() functions from this module. Use the functions from the e_errors module, with the same names, instead.
========== library_manager_client.py ====================================================================================
added operation for --vol command if state is ACTIVE
Fixed the help string for the --start-draining switch: 1) ingore => ignore 2) locked => lock 3) moved the list of values to the help string to preserve case sensitivity.
changed --vol output format
total time added to lib vols
========== udp_server.py ====================================================================================
Modified the __name__ == "__main__" test section to work with the udp_client test (for localhost) instead of being hardcoded for node 'happy'.
Cleaned up pychecker warnings about unused imports.
========== enstore_constants.py ====================================================================================
change NONE to ENONE
========== generic_client.py ====================================================================================
Re-raise KeyboardInterrupt and SystemExit exceptions from udp_client.send() in generic_client.send().