Time Series¶

Gravity Rift¶

Alert! The anti-matter reactor of the ship has become unstable. High-energy particles drift in random directions. You manage to avoid turning your ship into a fireball by throwing the anti-matter out through a hatch just in time. As it is the matter with anti-matter, it messes with the time-space continuum.

In this case, it creates a gravity rift that sucks you in. Your ship pops out in a completely different space and time, next to a habitable planet. Judging by the air pollution, you ended up in the early 21st century.

Fortunately, you should be able to make it back from this barbaric place. All you need to do is to find out when the gravity displacement is the biggest. Then, you should be able to open another rift and jump back to your own world.

You find the gravity displacement data over time in gravity.csv.

Use the local time system to calibrate the time.

Create a Time Series¶

Handling timestamps is one of the strongest features in pandas. Most of the time, you will want the timestamps to be in the row index. With pandas.date_range() you can create series of timestamps from scratch:

import pandas as pd

s = pd.Series(
    data = displacement,
    index = pd.date_range("2023-03-09", freq="D", periods=200),
    name = "reactor_temp",
)
s.head()

Instead of a pd.Series object you could also use a DataFrame, allowing for multiple columns. In both cases, a pandas.DateTimeIndex is used to track data over time.

You could also create timestamps specifying both boundaries and a number of periods:

pd.date_range("2023-03-09 08:22:00", "2023-03-09 16:00:00", periods=200),

Timestamps from Strings¶

Parsing strings to timestamps is very convenient. pandas understands plenty of different formats. This makes your life much easier, e.g. when parsing the log files of an anti-matter reactor.

pd.to_datetime(
    ["2020", "September 16th, 2020", "2020 Sep 16 11:11", "2020/09/16", "09/16/2020"],
    day_first=False
)

Technically, the timestamps are stored in integers and measured in nanoseconds since the Unix epoch on Jan 1st, 1970 when the first Unix machine officially started to tick.

Note

When reading timestamp columns from a CSV file, the option parse_dates=True is a useful shortcut if the timestamp is the index. Otherwise parse_dates=['col_name'] works as well.

Plotting Time Series¶

When plotting a time series, pandas automatically uses the time index for the x-axis:

s.plot()

A frequently used trick is to sum up values with a cumulative sum. The random data becomes a random walk. You see that the small changes add up over time and the data is drifting. Note that the resulting data still has a similar timestamp index:

ts = s.cumsum()
ts.plot()

pandas decides on the fly whiich scale and which ticks to use for the x-axis. This works well almost all the time:

Accessing DateTime Attributes¶

Every DateTimeIndex has a couple of useful fields that can be accessed:

ts.index.year
ts.index.month
ts.index.hour
ts.index.weekday
ts.index.minute
ts.index.month_name()
ts.index.day_name()

Indexing and Slicing¶

Timestamp indexes can be indexed and sliced comfortably using string intervals:

ts['1/10/2011']
ts["2023-03-20":"2023-04-17"]
ts[datetime(2011, 1, 7):]
ts['1/6/2011':'1/11/2011']

Resampling¶

A frequent task is changing rows so that different intervals between the time stamps are used. There are two types of resampling.

Downsampling condenses the data (fewer rows). Like with df.groupby(), you need to specify, how the rows should be aggregated:

ts.resample("1M").mean()
ts.resample("2W").sum()
ts.resample("10d").first()

Upsampling changes the index to a wider timescale (more rows). The resulting gaps need to be filled or interpolated, otherwise they stay empty:

ts.resample("6h10min").first()
ts.resample("6h10min3s").ffill()
ts.resample("6h").first().interpolate()

Rolling Mean¶

A frequent type of aggregation is the rolling mean (or moving average). It shifts a window of N data points over the time series and returns a value for each position. This smoothes out noise in the data.

ts.rolling(window=10).mean()
ts.rolling(window=10).std()

ts.rolling(window=10).mean().plot()

Try different window sizes and see how the curve becomes smoother and smoother. .rolling() also works for rolling standard deviations and custom functions.

Challenge¶

To find out when you need to jump back, solve the following tasks:

create timestamps for an entire year (2025), one per day
create a pd.Series with the gravity and the timestamps as index
display the weekday for each timestamp. On Sunday, your chef will be cooking bamboo with mushrooms, so no time travel on that day.
select the data after January. Time traveling before January is strictly forbidden.
calculate the sum of all gravity displacements for each of the remaining 11 months
calculate a rolling average with a 7-day window.
plot the rolling average data. The day with the highest average is where you should time-travel.