Trimmed Pipeline Stat Tracking

[TeachMeTW]

Optimize Pipeline by Streamlining Statistic Tracking

Summary

Introduces improvements to the statistics tracking pipeline by focusing on key metrics and removing unnecessary tracking for the bottom 80% and top 20% statistics. These changes aim to enhance performance and reduce resource overhead.

Changes

• Analyzed and reviewed bottom 80% and top 20% statistics to identify redundancies.
• Removed unnecessary tracking logic related to these stats from the pipeline.

Testing

• Verified that the removal of unnecessary tracking does not impact critical functionality by clearing pipeline and testing on an opcode.
• Validated the accuracy of remaining statistics and their integration into the pipeline by filtering today's pipeline stats.

Notes

From this test run.... Readings are in SECONDS

Top 20%:

data.name,data.reading
TRIP_SEGMENTATION,2537.513639458
TRIP_SEGMENTATION/segment_into_trips,2479.786787083
TRIP_SEGMENTATION/segment_into_trips_time/loop,2450.691215375
MODE_INFERENCE,1903.8646702920005

Bottom 80%:

data.name,data.reading
CLEAN_RESAMPLING,342.42430512500005
SECTION_SEGMENTATION,159.14454279100028
JUMP_SMOOTHING,64.29351804199996
TRIP_SEGMENTATION/create_places_and_trips,34.72458166600018
TRIP_SEGMENTATION/get_data_df,26.0028838125
TRIP_SEGMENTATION/segment_into_trips_time/get_filtered_points_pre_ts_diff_df,22.336031124999998
USER_INPUT_MATCH_INCOMING,0.41017545800000005
CREATE_COMPOSITE_OBJECTS,0.08192466699983925
USERCACHE,0.012076972333333344
CREATE_CONFIRMED_OBJECTS,0.010660459000064293
LABEL_INFERENCE,0.0088373339995087
ACCURACY_FILTERING,0.008748396000000325
EXPECTATION_POPULATION,0.006602458000088518
STORE_USER_STATS,0.00648329100022238

[shankari]

@TeachMeTW the problem with this analysis is that it doesn't take nesting into account.

To understand what that means, think about the high level of why we instrumented in the first place. If we remove the bottom 80% of stats, will it help us achieve that goal? Concretely, what is the next step after removing the bottom 80% of stats and will that take us closer to the eventual goal of improved scalability.

[TeachMeTW]

@TeachMeTW the problem with this analysis is that it doesn't take nesting into account.

To understand what that means, think about the high level of why we instrumented in the first place. If we remove the bottom 80% of stats, will it help us achieve that goal? Concretely, what is the next step after removing the bottom 80% of stats and will that take us closer to the eventual goal of improved scalability.

@shankari There was a 'problem' with the stage snapshot; when I was going through the top and bottom timings, there were some functions that does NOT exist in the recent master -- they might've been artifacts from the past.

I got rid of those functions that did exist for the bottom end and re ran the pipeline; now yes there is nesting -- the next step I am currently working is looking into the top 20% and seeing where its taking quite long and adding further instrumentation within those to zoom into the actual bottlenecks ie TRIP_SEGMENTATION/segment_into_trips_time/loop,2450.691215375 I am figuring out what is going in on that loop since it is general. After readding instrumentation within that nest, i will reanalyze and repeat until we reach a good point.

[TeachMeTW]

@shankari I’m still at conference and need to unpack the gear when I reach the hotel. Would it be possible to reschedule our meeting to tomorrow instead? Let me know if that works -- allows me to prepare better for the discussion as well

[TeachMeTW]

From Stage Snapshot

data.name,data.reading
SECTION_SEGMENTATION,163.54278820200125
TRIP_SEGMENTATION/segment_into_trips,57.53737983398605
TRIP_SEGMENTATION/segment_into_trips_dist/loop,49.804837625008076
JUMP_SMOOTHING,30.636910121160327
TRIP_SEGMENTATION,25.880560063461115
TRIP_SEGMENTATION/segment_into_trips_dist/get_filtered_points_df,5.787041137053166

data.name,data.reading
CREATE_COMPOSITE_OBJECTS,3.6322483392354474
TRIP_SEGMENTATION/segment_into_trips_dist/get_transition_df,1.9105706060072407
SECTION_SEGMENTATION,1.8852880927218085
JUMP_SMOOTHING,0.6412599327836848
TRIP_SEGMENTATION/get_data_df,0.5986348949954845
CLEAN_RESAMPLING,0.534879691346095
USER_INPUT_MATCH_INCOMING,0.49883880452567214
USERCACHE,0.3766976238503874
CREATE_CONFIRMED_OBJECTS,0.18336677490725026
GENERATE_STORE_AND_RANGE,0.07890600245445967
LABEL_INFERENCE,0.0739745471345357
EXPECTATION_POPULATION,0.03875759826421048
ACCURACY_FILTERING,0.02714171260689316
STORE_USER_STATS,0.020052012987434864
TRIP_SEGMENTATION/segment_into_trips_dist/post_loop,0.006629134004469961
TRIP_SEGMENTATION/get_time_range_for_segmentation,0.006610443000681698
TRIP_SEGMENTATION/segment_into_trips_dist/has_trip_ended,0.006048360512269555
TRIP_SEGMENTATION/segment_into_trips_dist/check_transitions_post_loop,0.002269156975671649
TRIP_SEGMENTATION/get_filters_in_df,0.0009775360231287777
TRIP_SEGMENTATION/handle_out_of_order_points,0.0005618800059892237
TRIP_SEGMENTATION/segment_into_trips_dist/mark_valid,0.00041384698124602437
TRIP_SEGMENTATION/get_time_series,1.821498153731227e-05
TRIP_SEGMENTATION/create_time_filter,3.462017048150301e-06
TRIP_SEGMENTATION/create_dist_filter,3.4130061976611614e-06
TRIP_SEGMENTATION/segment_into_trips_dist/continue_just_ended,3.3219694159924984e-06
TRIP_SEGMENTATION/segment_into_trips_dist/set_new_trip_start_point,1.0680523701012135e-06

I noticed that dist filter was only called... where is time? I am testing locally with an ios and android opcode with caebikes now.

[TeachMeTW]

How I am testing

1. Get an Android user and an iOS user via:
   db.Stage_Profiles.find({"curr_platform": "android"})

2. Find the username/opcode related to each user.

3. Clear the pipeline:
   ./e-mission-py.bash bin/reset_pipeline.py -a

4. Run intake on both users:
   ./e-mission-py.bash bin/debug/intake_single_user.py -e {Opcode}

5. Analyze the upper 20% and bottom 80% readings.

I realize that they are not similar—some users have more trips and other variations, so the readings are not accurate.

[shankari]

@TeachMeTW this #994 (comment) approach was good for a first pass. However, the point of adding the instrumentation was so that you could see the time taken when the pipeline was running on the server. You should use the stage_stats mongodump directly.

You can also bin by different types of users (e.g. based on duration of data collection or number of trips). The idea is that, with enough observations, little noise/variability fades into noise. If there is a clear signal in there, you can analyze groups of users separately.

[TeachMeTW]

@shankari I binned the stage trips by duration and this is what I found:

I found it weird how some took over a week.
The steps I took:

I analyzed the profiles in the db and took a look at the pipeline range.
Does the pipeline range mean when it was running the pipeline?
Some have values like this: {'start_ts': 1688576440.999893, 'end_ts': 1695177591.0009618} which mean:

Start Date: 2023-07-05 13:00:40.999893
End Date: 2023-09-19 22:39:51.000962

which seems quite long

[TeachMeTW]

I did more analysis; seems like its platform independent

curr_platform      android  ios
duration_category              
0-5 Minutes              0    0
5-15 Minutes             3    0
15-30 Minutes            1    0
30-60 Minutes            0    0
1-6 Hours                2    1
6-12 Hours               2    1
12-24 Hours              0    0
1 Day                    2    1
1-2 Days                 0    1
2-3 Days                 0    0
3-5 Days                 1    1
5 Days - 1 Week          3    1
1 Week+                 20   23

[TeachMeTW]

Going to pick an android and ios user from 1-6 Hour range to test pipeline again in stage

[TeachMeTW]

Tried the 1-6 Hour and 1 Week opcodes but did not prove useful; trying with larger times

[shankari]

Before rushing off, you might want to verify this:

Does the pipeline range mean when it was running the pipeline?

If that is not true, the rest of your analysis is invalid.

Note that you can't expect an answer from me for questions in the middle of the day (or night).
Slow down and work methodically.

[TeachMeTW]

I took a look at the profiles, and looked at last update ts; latest was in October so I began looking at pipeline_time instead to see the latest updates to 'STORE_USER_STATS' indicating it was ran when the latest was merged

[shankari]

@TeachMeTW I would suggest not focusing on what you can do, but what you want to achieve. What are you trying to figure out? What is the overall goal?

[TeachMeTW]

@TeachMeTW I would suggest not focusing on what you can do, but what you want to achieve. What are you trying to figure out? What is the overall goal?

I am trying to figure out if both an IOS and an ANDROID was put into the pipeline during Nov 8 when the staging was merged with the changes.

The main reasoning is that both use different filters -- one uses time and the other uses distance. I noticed on the mongodump that there was functions that doesn't exist anymore and functions that were never called.

I wanted to investigate if 1) both android and ios were tested, 2) Retest on stage op codes featuring an android and ios to see if I need to add more in depth instrumentation or remove some and see what areas need improvement

[TeachMeTW]

What I found was that only an IOS device was tested, meaning the Android was untested.