This section contains descriptions of the three quality control algorithms currently implemented by the Wind Profiler Demonstration Network's Hub software. The three algorithms are, in order of execution: 1. Consensus Averaging 2. Time-Height Median Check 3. Vertical Shear Check The following descriptions are intended to provide a general understanding of the algorithms. The details of the algorithms are available from the Profiler Program in the form of documented FORTRAN source code. CONSENSUS ALGORITHM This algorithm operates on the ten 6-minute radial velocities (i.e. lst moments) from the profiler radar for a given hour. For each gate of each beam (east, north, vertical), the ten 6-minute radial velocities are examined to determine whether they are within a predetermined delta velocity of each other. The number of velocities which are within this delta velocity is called the consensus number. The consensus is said to have been passed when the consensus number is greater than or equal to 4. If the consensus is passed, those velocities which comprise the consensus are then summed and divided by the consensus number. This results in a consensus-averaged velocity. In addition to determining the consensus-averaged velocity, a significant portion of the algorithm determines for the vertical velocity. This is done differently for the Low and High Modes as described below. HIGH MODE The consensus number is determined for the ease, north and vertical beams' radial velocities. CONSENSUS PASSED: The 0th, 1st and 2nd moments of the 6-minute periods which comprise the consensus-averaged data are summed and divided by the consensus number. In addition, the correction for the antenna elevation is made by dividing by the cosine of the antenna elevation angle. It is not necessary to have a consensus on the vertical beam in order to produce a wind in the high mode. CONSENSUS FAILED: If the consensus number is less than 4, then an indicator is set for the appropriate gate indicating that no wind could be produced, i.e. there was no consensus. CORRECTION OF WIND COMPONENTS FOR VERTICAL VELOCITY: A correction is made to the U and V wind components for the vertical velocity of the wind at a particular gate in the event that all of the following are true: 1. A wind was produced, i.e. both east and north beams passed consensus. 2. Vertical velocity passed consensus. 3. Absolute value of consensus-averaged vertical velocity is greater than 0.15 m/s. LOW MODE The Low mode consensus processing is more complicated than that of the high mode. This was done in an attempt to properly correct the wind components for the vertical velocity in the presence of precipitation. The consensus number of the vertical beam is determined. If the consensus number is greater than or equal to 4, the 0th, 1st and 2nd moments for the vertical Low are summed and divided by the consensus number. The consensus numbers of the east and north beams are determined. Then each 6-minute radial velocity of these two beams is corrected for the vertical velocity using the vertical radial velocity for the same 6-minute period. The absolute value of the vertical velocity must be greater than 0.15 m/s. The correction is as follows: where: Vel_radial = east or north 6-minute radial velocity W_radial = vertical 6-minute radial velocity Then the consensus numbers for the corrected velocities for the east and north beams are determined. At this point, there are 4 consensus numbers: east and north uncorrected,east and north corrected. It is then left to determine which (corrected or uncorrected velocities will be used to calculate the hourly averaged value for the 1st moments. The 0th and 2nd moments consensus averages will be determined by using the consensus number that corresponds to the method that has the higher minimum consensus number for the oblique beams. For example, given the following consensus numbers: Uncorrected east, Uncorrected north: 7, 8. Corrected east, Corrected north: 6, 9. Uncorrected radial velocities would be used. Although the north beam has a higher consensus number using the corrected method, the minimum consensus number for the uncorrected velocities ( = 7) is greater than the minimum consensus number for the corrected velocities ( = 6). NO WIND PRODUCED: For no wind to be produced in the low mode, one of the following must be true: 1. There is no vertical consensus. 2. One or both of the oblique beams did not pass consensus for both methods, i.e. need to pass consensus on both beams for at least one method. TIME-HEIGHT MEDIAN CHECK This algorithm was developed by Keith Brewster of PROFS. See the document, PROFS Operational Profiler Quality Control 24 April 1987 revision for the details (Attachment 1). The WPDN HUB implements the identical algorithm except for the following minor difference: In the high mode, 3 gates above and 3 gates below the current gate for each hour are used in the calculation of the median value. This was done in order to take into account the redundancy in the high mode range gates. Attachment 1 The PROFS operational profiler quality control system (also referred to as "combined quality control") is a multi-stp process, consisting of modules run by the Wave Propagation Lab (WPL) and PROFS. Currently, the processing is done for specific workstation products and data are not stored in the PROFS database with the quality control flags (except for the WPL consensus averaging). The quality control is designed to detect spurious values and does not in any way filter out statistical or small-scale meteorological "noise". This also means that erroneous data that lie within normal statistical noise of the data will be retained. Order of processing: 1. Consensus Averaging (WPL) 2. Time-Height Median Check 3. Vertical Shear Check on each mode individually 4. Vertical Shear Check on combined profile 1. Consensus Averaging The consensus averaging is done by the WPL profiler data processing system that produces the horizontal winds from the radial velocities of the three profiler beams. The samples of radial velocities for each beam acquired during the hour (ten samples per beam per hour) are assembled and those that agree within a preset threshold (s meters per second) are averaged to form a consensus radial velocity average. The radial velocity averages are then geometrically combined to form the horizontal wind direction and speed. Should there be less than four samples from the ten that meet the agreement criterion, no average radial velocity is computed, and if there are any radial velocities missing no horizontal wind is computed. In other words there must be a consensus on each of the beams at that particular gate or a horizontal wind is not computed. These consensus averaged values are the data are stored in the PROFS databases and are the raw data that are used for the subsequent quality control (for workstation products) or forwarded to other users. 2. Time-Height Median Check The median check consists of several steps. First data are assembled from two previous hours and the current hour. Then, for each data point in the "current hour", the program searches for the eight points "surrounding" it in time and height. The median of theses values (done separately for each wind component) is compared to the observation. If the median and the observation differ by more than a threshold, the median is recomputed using just the last hour of data (not the two previous hours). If there still is disagreement, the observation is flagged bad. Median Check Threshold Threshold = a h2 + b h + c where a = -7.89 x 10-8 s-1m-1 b = 1.54 x 10-3 s-1 c = 9.50 ms-1 Threshold is in meters/s and h is height above sea level (m). This is a quadratic fit to the values: threshold height ASL 9.5 ms-1 0 m 17.0 ms-1 9000 m 14.0 ms-1 16000 m and was designed for use from 2000 m ASL to 18000 m ASL. Threshold based on speed: threshold = 0.4 * spd, where spd is the average of the observed wind component and the component determined by the median The threshold used is the maximum of these two methods. The threshold is increased if the data from previous hours are from several hours ago. The average time of the points used in the median computation is found and issued to adjust the threshold. Factor to multiply threshold by based on time difference factor = 1 + 0.18 (DT - 1.) where DT is time difference in hours between data and median points except when that difference is less than 1 hr. Then, DT = 1. The threshold outlined above is quite large as this step is currently used just to identify those points which are extreme outliers, particularly those caused by precipitation contamination in one or more of the beams. The idea is to prevent the discarding of dramatic winds shifts in time caused by strong short-waves or fronts; the incidence of such erroneous flagging is extremely small. Future enhancements New processing performed by the HUBOPS computer is sending PROFS the number of samples which "agreed" to form the consensus average for each gate. These numbers will be used to further refine the threshold. Smaller numbers of agreeing samples will result in a tighter threshold. We will also look at newly available reflectivity data to see if they will be useful. Exceptionally high (perhaps caused by solid target like birds or by precipitation) or low reflectivities could be used to decrease the threshold. 3 and 4. Vertical Wind Shear Check This step is performed twice on each wind profile. First it is applied to each mode of data separately (there are two or three operating modes at each site), then its applied to a "combined" profile consisting of all modes meshed together. The program begins from the lowest reported level and steps upward. At each point the full wind vector shear (between a "lower" gate and the gate being tested -- the "upper" gate) is computed and compared to a threshold. Establishing the threshold is complicated, allowing for: 1. Differences in expected shear with wind velocity 2. Apparent shear caused by statistical uncertainty in the wind measurements for two closely spaced observations 3. Increased shear expected in the vicinity of the jet core Threshold calculations a. Vector difference threshold: 10 m/s b. Vector difference threshold based on wind speeds: 0.42 * spd, where spd is average wind speed of the tow gates vecthresh is maximum of a and b. c. Gate-to-Gate shear threshold: 0.018 (m/s)/m d. Vector difference shear threshold: vecthresh/600 600 is chosen to make a approx equal to c e. "Random error" shear threshold: 7 9m/s) / dh where dh is height separation between obs shear threshold is maximum of c, d, and e above and is used to compare to the vector shear between the two gates Increase in threshold for the case of unidirectional shear to allow for more shear in the vicinity of the jet: shear threshold = unifactor * shear threshold unifactor = 2 if the directional difference between the gates is less than 20 degrees unifactor = 0.5 if the directional difference between the gates is greater than 70 degrees unifactor = 1 for all other cases Should the vector shear between gates exceed that computed by the above method, the winds are checked ut by each component. The gate-to-gate component differences are calculated and the difference between the upper gate and the next gate is checked. If the ratio of the first difference (upper-lower) and the second difference (next-upper) is greater than -2.0 and less than -0.5 (i.e., approximately -1) then it is assumed that the upper gate is bad, and it is flagged. (Note: if one component meets the criteria, the entire observation is flagged "bad"). Once a gate is flagged, it is NOT used for checking other gates above it. If the ratio criteria are not met, tow possibilities exist: 1) there is a frontal zone or source of large scale shear between the two gates, 2) there are two adjacent bad winds (the upper gate and the next gate). To determine if case 1 is true the large sale shear is examined by fitting a line (to the wind component) between the lower gate and the component observation three gates above it. The value determined from interpolating that line to the height of the upper gate and to the next gate (i.e., two gates above the lower gate) are compared to the observed values at those gates. If the tow observed values differ from their interpolated counterparts by more than 10 meters per second and the difference for both is in the same direction (i.e., both greater or less than the interpolated values), they are both flagged "bad". Special processing is done if shear exceeds the threshold and the lower gate is the first gate with good data. In this case the wind components from the next two gates are the extrapolated linearly down to the height of the first gate. If the extrapolated value is reasonable, it is compared to the observation. If the difference between the two values is greater than 8.75 meters/sec, the first gate is flagged bad. This test is only done for "meshed" profiles, with the assumption that a better test of whether the observation is good or bad can be done after meshing (i.e. the point may no longer be an end point). There is also special handling for the top data level (again only for "meshed" profiles). Here, the same extrapolation and comparison as in the case of the lowest gate is done, and the threshold used is also 8.75 meters/sec. Future Enhancements As with the median check (see Section 2) the newly acquired consensus numbers and the reflectivities may be used to further refine the threshold and/or to refine the decision process when it is difficult to discern which of the winds causing extreme shear should be flagged. The increase in thresholds to allow for shear near jets has caused a few problems of bad data being accepted aloft where dh is fairly large. Further refinement of the vecthresh parameter may be required. Detection of spurious values in the first few gates of the profile caused by electronic noise in the antenna and receiver has proved to be very difficult. In face the majority of spurious values that the system fails to recognize are there noise contaminated winds. This problem could be attacked in several ways. The consensus numbers and reflectivities may provide more evidence, and a method of "blacklisting" could be imposed that would decrease the threshold for those gates know to be chronically bad (this would introduce a site-dependent condition on the processing-- something that has been avoided thus far). Finally, a better method of extrapolation of data from the next higher gates where the noise affect is considerably less, will be tried. This extrapolation could also be converted to an interpolation between the profiler data and surface data. Surface observations taken at the profiler site would be most useful in this effort, but lacking that, standard surface observations could be employed. Using standard surface observations would have the drawback of introducing another data source, which itself could have errors and may be missing at times. VERTICAL SHEAR CHECK ALGORITHM This algorithm was developed by Keith Brewster of PROFS. The algorithm is detailed in Attachment 1. The WPDN Hub has implemented the algorithm as developed except for two minor changes which were desirable due to the differences between the network 404 MHz profilers and the profilers for which the algorithm was originally developed. The changes were made by Jack Harlan and Russ Chadwick after discussions with Keith Brewster. The changes are as follows: 1. The algorithm is performed on each mode separately. There is no combined-mode checking. 2. The precision of the radar, 0.1 m/s for radial velocities, is taken into account when any velocity thresholds are used to determine whether a wind vector passes the shear check. In addition, 3 of the constants in Attachment 1 had been updated in the source code that Keith Brewster provided to the Profiler Program. They are as follows: Under "Threshold calculations "a", 10 m/s was changed to 9 m/s Under "Threshold calculations "b", 0.42 was changed to 0.4 Under "Threshold calculations "c', 0.018 was changed to 0.016 Preliminary Checking: the data for each mode is selected such that all gates which have failed either the consensus or the time-height median check are eliminated. However, gates which did not have enough surrounding data points to perform the median check are retained. General Overview: Each mode is traversed from the lowest gate to the highest. The first gate is assumed correct and the next gate is compared to it. If the second gate is found to be "failed", then an extrapolation is made using the slope between the third and fourth gates and the height difference between the first and second gates. If the second gate is then determined to be correct, the first gate is flagged as "failed". This is continued to the top of the mode or the last gate in the mode for which there is data. Both the vector difference and the slope are calculated using the current gate and the gate below )the gate below is checked to see if it passed the shear check). If either of these quantities is greater than a calculated threshold, then the gate is flagged "failed". In the case of a failure, an interpolation is made between the gate below and the current gate. The differences are checked against a constant threshold and if greater, the gate is flagged as "failed". If the differences are smaller than another constant threshold, the gate's flag is changed to "passed" Step-By-Step: Calculate the Slope between Gates for each Wind Component 1. Find u, v component differences between current gate and previous one: UDIFF, VDIFF 2. Find height difference between current gate and previous one 3. Divide each component difference by the height difference - UPRIME, VPRIME 4. Find magnitude of slope: SQRT(UPRIME**2 + VPRIME**2). Calculate the Vector Difference between Gates 5. Find magnitude of vector difference: SQRT(UDIFF**2 + VDIFF**2) Determine whether Unidirectional Shear Condition 6. Find directional difference between previous gate and one ABOVE current gate 7. If directional difference < 20 degrees: a. Calculate mean direction using previous gate and one above current gate = DIRMEAN. b. Find difference between DIRMEAN and direction of current gate = DIRDIFF. 8. Else if directional difference >+ 20 and previous gate is not gate #1: a. Find directional difference between current gate and one above = DIRDIFF. 9. Else if directional difference >= 20 and previous gate is gate #1: a. Find directional difference between current gate and gate #1 = DIRDIFF. b. DIRDIFF = maximum of 21.0 and DIRDIFF 10. If DIFDIFF =< 20 degrees: a. Shear_factor = 2.0. Allows for greater differences due to unidirectional shear 11. If DIFDIFF >= 60 degrees: a. Shear_factor = 0.5 12. If DIFDIFF between 20 and 60: a. Shear_factor = 1.0 Calculate Thresholds for Vector Differences 13a. Multiply the mean speed of current gate and previous gate by 0.4, empirically determined constant. b. Vector Threshold = shear_factor * MAX(9.0, result of 13a.) in meters/second. Calculate Thresholds for Slop Differences 14a. Divide 13a. by 600 b. Divide "7" by height difference of current and previous gates c. Slop Threshold = shear_factor * MAX(14a,14b,0.16) Perform the Shear Check 15. If Slope (Step 4) is greater than Slope Threshold or vector difference (Step 5) is greater than Vector Threshold, then gate fails shear check. 16. If a gate fails and the height difference between current gate and previous gate is greater than 250 meters, then perform the following interpolation: a. Find slope between previous gate and 2 gates above current for each component. b. Find difference between current gate and previous gate interpolated to current gate height. c. Interpolate previous gate's component to one gate above current gate using slop calculated in 16a. Do this for each component. d. Square each value from 16b. e. Square each value from 16c. f. If 16b > 81 (m/s)**2 and 16e. > 81 (m/s)**2 and (16c./16b) > 0, then gate still fails. g. Else if 16d. and 16e. are each < 16 (m/s)**2 then gate passes shear check.