Transcript
This transcript was autogenerated. To make changes, submit a PR.
Hi, welcome to this talk on the mythical machine learning
pipeline. My name's Jim Dowling, and I'm going to talk about
how we can refactor this monolithic machine learning pipeline that many
of you may have heard about before into what we call
feature training and inference pipelines. We're going to do it all in Python,
and we're going to decompose the machine learning pipelines into
these smaller, more manageable parts to build machine learning systems.
So I'm going to start by making a claim that mlops, or machine
learning operations as it exists today, is too hard. It's kind
of like this telecom tower here. It's a very brittle set
of systems that we're plugging together to try and put machine learning at production.
But really it's too complex to make anything which is
going to be either easy for developers and in particular python
developers to put in production or for people to
maintain. So if we look at mlops according to Google on
this diagram, we can see here a lot of boxes. So this notion that you
start at data and you go through many stages of processing the
data, validating it, preparing it, training models, validating models,
and all of this happens in one big, monolithic,
orchestrated ML pipeline. I'm going to show you that
we don't need to do it that way. Databricks are following a similar
pattern for how to encourage people to what they
say is follow Mlox best practice.
I see this really as being an overcomplication of something which is much
easier to do, which is just to build machine learning systems
and do it to the principles of decomposing complexity and
then putting those parts together again. And for ML ops
in particular, which is really about building systems
that we can incrementally improve and automatically test
and version and so on, we don't need to go down to the infrastructure
level, we can stay in Python. That's one of the key points I want to
make here. You don't need to become a dockers or a Kubernetes expert
in order to do ML ops.
This is classic tensorflow extended. This is what
many people learn in courses that they take in machine learning operations,
that you need to go from the very beginning to the very end in one
big m to ML pipeline. So machine learning pipeline,
I say it's mythical, because in practice, I've never seen
an end to end pipeline written like this. Where the data comes
in, it's validated, transformed into features,
models are trained, models are analyzed, then they're validated,
and then they're served all in one big monolithic directed
acyclic graph. That's not typically the way it works. And the
reason why it doesn't work like that is because monolithic ML
pipelines, they couple different natural stages
of machine learning systems. So you have what's called
feature engineering, where we take the raw data and turn it into the data,
the features that we're going to use to train models, but also
to make predictions with. Now, if you couple that phase
with the model training phase, training might require
gpus. If you're using deep learning, feature engineering does not require
gpus. You put it all into one large system,
and then suddenly you're using maybe gpus for feature engineering,
which is quite wasteful in terms of resources. If your feature engineering
is done once a day, because your new data arrives every day, but you
need to predict every hour, why would you couple those two things together?
Inference should not be coupled to feature engineering when they run at
different cadences. By coupling them all together, you're adding development
and operational complexity. You're also not reusing any
of the data or features that have been created by your
feature pipelines, and it's too hard to build
these systems in such a way to get to a minimal viable product.
I'll give you an example of one, Kubeflow pipelines, just to pick on one
could have picked on many different systems. They claim on their website
that it enables you to reuse components and pipelines
quickly, easily, to create end to end solutions
without having to rebuild each time. I've never seen this happen in practice.
So mlops, we still want to follow
the principles of mlops, and that means testing your software.
And you can think of it as being a hierarchy of needs. You have raw
data which needs to be tested to create features. Features are
used to create models, so the features need to be tested in order to create
the models, and the models need to be tested in
order to use them by the ML enabled applications. And typically your ML
enabled application will want to a b test a model before it then uses
and switches over to a new version of a model. But that's where we want
to get on the top of the pyramid of needs, if we will,
for mlops. But we need to start somewhere. And the place
to start, as any software developer will tell you, is with
a working system. And in this case we're building a machine learning system. So we
need to get to a working machine learning system as quickly as possible.
And that means we need to have some code which will create the features
we need to have some code to train the model, and we need to have
some code to make predictions or inference on new data that
arrives using our model. So ML
Ops as a set of principles helps you to get through
a working system as quickly as possible with a baseline and
iteratively improve it. And the reason why you should be able to iteratively improve
your software, if you're following DevOps or ML Ops principles,
is that you're testing. We're testing the features, the models,
and we're versioning the features and models. So if we're going to do upgrades,
if we're going to deploy a new model that's connected to some new features,
if they're both not versioned, we'll have a terrible time
connecting those two things together and ensuring that things can be safely upgraded.
So once you have testing and versioning in place for
the two main assets that we see in machine learning systems, features and models,
then you're able to move quickly, you're able to make small changes
and improve your iteration speed, and you're testing your
models and you're testing your features. So you're improving the quality of your software,
and that's ultimately where you want to get to where you can move more quickly,
make small changes to iteratively improve your systems,
and be confident that the changes that you make will not break everything.
So that's the goal of Mlops, and that's what we as developers would
like to use to make our code better. Let's jump
into ML Ops. In Python, we're not going to use infrastructure.
There'll be no docker or Kubernetes or we're not
going to talk cloud infrastructure. We're going to talk about the way
in which we advocate, and particularly I advocate for structuring these
machine learning systems to make it easier to manage them.
So what we typically have are these three main components, the feature pipeline,
the training pipeline, the inference pipeline. I call this the FTI
pattern to make it easier to remember you have one program. In this
case, we're looking to look at Python programs, which takes data from our
data sources. This program will be typically an operational program,
so it might need to be scheduled. It could be airflow if it's Python,
or it could be a Python program that's scheduled to run
in the cloud. So there's some pretty nice tooling out there, like modal,
who allow you to schedule these Python programs
with a cron like hourly run or daily
run. And those programs are basically going to read your
raw data, compute, features. If you've got supervised
machine learning, you may need to create labels as well. And rather than
store that data in an object store or in a distributed file
system, we're going to look at a feature store as a way
to store that data. And that's because the feature store will provide
us with a very nice data frame API. Much feature engineering and feature
pipelines are written in frameworks like pandas. If you need more performance,
you might move to polars. If you have a lot of data, you might move
to Pyspark. But the output of all of
those is a data frame. So if you can write that data frame, and you
can obviously do that in a secure manner. So we'll use API keys.
For example, when we write to Hopsworks, then you don't
have to worry about the complexity of if you're in a cloud environment,
getting access to a particular bucket, an IAM role that gives
you permissions to do that. It's basically going to be an
API key, some privileges associated with it, and then write
your data frame. So your training pipeline should be able to use the
feature store in the same way. We'll be able to say, okay,
I'm going to select features from all of the available features,
create some training data, select labels as well, train my
model, and when my model has been trained, I'll need to store it somewhere.
So typically where we would store models is in some place called a model registry.
There are many model registries out there, some of them are hosted
over the Internet. Hopsteryx is also hosted over the Internet.
We can call that serverless model registry or serverless feature store.
But it basically means at least when you're getting started, you can write
a python program on your laptop or in colab, and it
can read from a feature store and write your model back into a model
registry. So Hopsrix also is a model registry. We'll look at that later.
And then finally, when you have an inference pipeline, this is when you want to
generate value for your model. And this is where most machine learning courses
stop. They'll train a model they'll evaluate on a static
test set. So a holdout data set to see how the
model generalizes on data it hasn't seen before, which is
great. But if you want to see your model in the natural environment where
it's creating value, you should write an end to end system
so that new data can come in through the feature pipelines. And for a batch
inference application it can take that new data that's been written via
the feature pipeline. Read it up as features, read up the model from the
model registry, generate the predictions, and store them somewhere
where maybe a downstream dashboard or an operational
system will consume those production to make those applications
AI enabled. Of course, this generates logs. You typically want to save
those to help improve your observability and monitoring of the system.
So I'm going to look at a little demo later on. The code is available
on the link below. It's basically
looking at credit scoring as a machine learning system. This is quite a popular
type of machine learning system we see at financial
institutions. You may have information about people
who would like to apply for credit, and we'd like to give them a score
to decide on whether we're going to give that person credit or
not. So typically what you'll need to have is data
from different sources. And you can see there's some sources on the left
here. A feature pipeline will create the features from those
data sources. Training pipeline will train your model. We look
at an Xgboost model and then inference pipeline will take
this XgBoost model and then some new credits.
So credit applications to score and that could be done in a batch manner.
You could have like a batch that arrive and once a day you score them
and then you send an email out the next day to say you're approved.
But even better would be an interactive application. So as the user goes
to a website and fills in their details, then we can have an online
system which can read those features back and the feature store and
hopsearch in particular enables that. We're going to look at the batch case
today and the training pipeline. It can
be run in any kind of training environment, any Python
enabled environment. And our pipelines can also be run in any
Python environment. So today we'll look at running it in my notebook,
but of course you can schedule them to run in any Python
environment. So let's start and look at feature pipelines
and have a look at a little bit of the code that we need to
create a feature pipeline. So the feature pipeline will take your raw data,
it'll compute features from it that's compressed signal that we're going to use to train
our models with and also to make predictions with. So there's an example here
we're using pandas to do our transformations. So we
call these model independent transformations because the features we store in the
features store should be reusable across many different models.
We can see here that here we're doing a classic aggregation
where we're counting the number of events happening in a four hour window.
We can see here that when we've created
a data frame window AGSDF here with our features
that we'd like to use in our feature store, we'd like to
store them in the feature store. And these features here are related to the credit
card, the number of transactions, or the frequency of the transactions in
that four hour period of time. But basically with this
window AGS data frame, what we're going to do is we're going
to insert it into this thing called a feature store. So here we're creating the
feature store. We're giving it a name and a version, a description.
We're identifying which column is the primary key. So the unique row level identifier
column that uniquely identifies each row in this data frame.
If there is a unique timestamp column
within that particular data
frame, it doesn't need to be unique. Of course, they can be
the same across many different rows, but if there is an event time at which
that particular row was generated as a column, you can specify
it here, because what we'll see is that in feature stores,
if I have many different tables of features that have different event times on them,
we need to line those up so that we don't get what we call data
leakage. So we don't get future data. When we join these columns
of features together from different tables, we don't want to have data that's
in the future because then our model will be able to learn from future data
which we don't want to do. So that's called data leakage. We want to avoid
it. So if you specify the event time column in your feature group that
can be used by the feature store to do what's
called a point in time correct join, so there's no data leakage. So once
you've created this feature group object, you can just insert your pandas data frame.
So you just call insert on it, it'll write it to the feature store and
your data will end up there. Now,
Python is great for that smaller scale of data. If you have
larger volumes of data, you may want to use to Pyspark. It's great
for scale and testing, but it's obviously a bit more challenging to develop with,
more challenging to debug with and operate. SQL is also
quite popular. SQL has low operational
overhead. It can scale quite well. There are many different features, as you know,
that are very difficult to implement in SQL.
Even if you embed udfs in your data warehouse,
things like embeddings and there's many libraries
of course, in pandas we use to compute features that are not particularly suitable
in SQL. Now Python is obviously
a great and popular language to develop
features in, so it has low operational overhead.
Pandas is very popular, Polars has become popular.
It scales to larger data volumes than pandas,
but they're all good choices. Now if you need stream processing,
so very low latency features that are computed on real time data,
then you might want to look at a streaming framework like Flink or
Pyspark, which also has spark streaming.
So if you want to write that code in Python, probably Pyspark is your best.
But Flink is very Java centric.
So in hopsworks we can write the data
to the feature store from Python in what we call a streaming API
or a batch API. The batch API will store the data offline. It's only
historical data, so it'll only be available via this offline API.
So the offline API is to get training data or batch data
that you'd like to do inference on. But if you have an online application,
you need low latency access to your features. So if we
have for example the credit scoring application that needs to look up your
features within a few milliseconds because it's going to do a live prediction
of your credit score, then you need the online
API. So in that case you write using the streaming API.
Now in Hopster's the default is a streaming API.
So when I called insert on my feature group, it wrote the data via
this API. It'll be stored both an online offline store and
it'll be available via both the online and offline APIs.
Now what we can see here is that the tables of features
are called feature groups, and then we have something called a feature view.
So in hopsworks you're able to reuse features across many
different models. And the way you do that is by selecting features into something called
a feature view. So you can select from many different feature groups, all updated potentially
by different feature pipelines, and then the feature store will perform
this point in time correct join for you.
So if we're able to reuse features across many different models,
we can see here one model at the top it has a feature view selecting
a certain set of features, another feature view down here selecting different features,
and each one can create their own training data sets. Train your models on
and each one can pull out. Then if we're doing batch inference, you'll pull the
features by the feature view, say well, the data that arrived in the last 24
hours, for example, and then you'll
do inference on those with your model.
Now training pipelines use these feature views to
read data to train models with. So it
uses this offline API. You can get your data back as files,
maybe CSV or parquet or TF records
even. Or you can get in the back as a pandas data frame if the
data is not too large. And you can train directly with
your data. And it has if support written nice things like random splits and
time series splits of your data. So you don't need to do that. Even in
Scikitlearn you can just read the ready made split data
frames with your features and labels.
So the feature view itself is an API for model development and also
for operations. In the online case, you basically select
your features. You say what label it is for this particular feature
view. One other thing that you might want to do is features
are typically shared untransformed. So what that means
is if you have a categorical variable, store it in the feature store unencoded,
and then when your model is selected, it can encode it, because some models
will need to encode categorical variables.
So for example gradient descent based models, but other
models, so cat boost for example,
can work directly with the categorical data. And the same
is true for numerical variables. So we call those model specific transformations.
They can be associated with features in the feature view so that you don't have
to write that code separately yourself.
Because there is a potential that if you write the code in the training pipeline
to transform features like encode them
or normalize numerical features, you need to do the same thing in the
inference pipeline. And there is a potential for what we call SKU there.
So the feature view will help you avoid that potential problem,
feature view. Then once you have it, it can apply the model
specific transformations on the data, create training data, batch inference data,
and it'll apply those transformations consistently.
So the point in time correct join I mentioned already, you take tables from
features from different tables, ensure that they're lined up on
the correct transaction event time here. So this value
of this category at this point
in time, we can look up the amount
for that particular week and month, and we're not going to get future values in
here. There'd be no data leakage. Now, if you
were to write this join this point in time correct join yourself. It's going to
be quite complex. For that example, we can see there's quite a lot of SQL
code for it, but in Python you just write the following code below.
You select the features that you'd like from the different feature groups.
You join them together in pandas like syntax,
and you then get back what's called a query
object. This is our selection of features, and then we can create our
feature view with that particular selection of features.
Now once we've created our feature view, we can see we've got a feature view
object here, we can create training data from it, and we can
ready split that train data into random sets, a test
set and a training set, and a validation set as well, even, and you
can say what the file format you'd like to store that train data is.
You can also get the train data back directly as pandas data
frames. So here we can get our features in the training,
test and validation sets and the labels in the training,
validation and test sets back. And I don't need to run it through scikitlearn
splitting algorithm. This is going to be a random split, but there is also a
time series split. So you can say if I've got a
time series model and time series data, I'll train on maybe
the data here up to version two, and then I'll predict
how my test set will be the data that came after the
end of this particular point in time.
Now what you can also do with a feature view is you can get training
data, for example for version one of your model, or get training data for version
two. New data will keep training in the system, the feature pipelines will keep
training new features, and then you can read that
data and make predictions on it. So we can do batch inference with
the feature view as well. We create batch inference data and
finally for feature views we can specify specific features
that we'd like to transform. So here we're saying the amount of money last month
we're going to apply the standard scalar, the category
variable. We're going to encode it, it's categorical variable,
apply a label encoder on it, and the same for amount last week,
the standard scalar. So when we read the training data, the batch
inference data, or in this case the online inference data, the feature
vector, it'll apply these transformations after it's read the
data in the feature store, but before it's returned to the client. And this ensures
consistency of these transformation functions between this training pipeline
and this inference pipeline. You can do it yourself in scikitlearn pipelines,
and then you need to make sure that both of these are consistent.
So let's move over to inference pipelines and we'll
have a look at some of the code. This is a predictor
class. So this is a model that you will be deployed in a model
serving server. We can see here the method called predict.
But before we predict, we need to initialize this object. So when it's
loaded into our model serving server, init will be called
we'll connect to the feature store that we can see here. We'll initialize the feature
store. And what we need to do is we need to tell it what
the version of the training data was that we trained
our model on, because we can create many different train data sets from a feature
view. So here it was version number one. So we say, okay, I'm going to
initialize my feature view on version number one. The reason we need to do that
is the transformations often need the state from
that training data set version. So if I'm normalizing a numerical feature,
I'll need to know what the mean value of that feature was in
the training data set the model was training on. So that
information is captured in the feature view. You don't need to supply train data set
when you're deploying your model for serving here.
And then you can see we're loading the model from the model registry and it's
now available in our predict object. So when a request comes in to make a
prediction, the model is loaded, the connection to the feature store has been established,
and we can just call self dot model
predict, return the pre computed features from the
feature store here and then send them to the model to make the prediction.
So this is going to go to the feature store, return our precomputed features.
We're going to cast them to a numpy array, and then the
model predict method will be applied to that numpy array to make our prediction,
which is returned to the client. Let's have
a quick look at a demo. So this
particular demo, the way that we often see
a lot of these machine learning systems built when
people are starting out, when you want to use get started as
quickly as possible, is you can write Python programs in a very
nice tool called modal. So modal has a very generous
free tier. It basically can schedule Python programs
for you. And if you need to install libraries, you just annotate the functions
and say Pip, install hopsworks, for example.
And in that modal code we can write our feature engineering pipeline that we
saw earlier. It'll read the data and write data frames to hopsworks.
And then modal can also train. You may want to train because training is
not an operational system. You can train on colab and do it
offline, if you will, or on demand, or when the
model has become stale. You don't necessarily need to do it every day as you
would when in a feature pipeline if new data was arriving every day,
and then for inference. Again, it's an operational system.
So maybe it's going to be either on demand when the user
goes into the website in an interactive system, makes a prediction,
or maybe it's going to be a dashboard that gets updated
at some cadence. So hugging faces is a really nice way of
doing a lot of inference, because it has, again, a very generous
free tier where you can use hugging face spaces to
do some really nice uis for dashboards or for interactive applications,
in this case from hopsearch. They can read the models and the
data frames. This is the data that you want to make your predictions with.
And we can also even save the logs back to hopsearch
and even do user interfaces in hugging
face for monitoring our models in production. So hopsearch in this
case, again, has another free serverless tier.
You don't need to install anything, you get 25gb of free storage.
This is a really nice way to build what we call serverless machine learning
systems. And here's some examples. There's a bunch of these on the
Internet. I think there's about 40 or 50 of them now. Some really nice ones
are air quality predictions. In Poland, this is a streamlet application.
It runs the feature pipelines daily on mobile,
and it stores then these features and hopsworks. And then
when you go to the dashboard here, it can redraw with the
predictions of the air quality in the different cities. So this is for
today in early, this is actually early March,
and you can look at different days and it will connect to hopsworks, read down
new data to score and redraw this particular UI
and streamlit. And there's another one here that will predict whether
your post on Reddit will be liked or not. So it's using sentiment analysis
and it's very interesting. Another one on Tesla, stock price production.
So I think it's using sentiment from Twitter, amongst other
places, and then another one using New York electricity.
A lot of services globally have been digitalized and a lot of data is
available now for us to build these prediction services with.
So you can write a Python program which will go to the
New York electricity market. And if you follow this link,
you'll see where they get the data from. And you can see in many
cases, it's actually outperforming the ETA's forecast or
the EIA's daily forecast, which it's doing here.
So there's a lot of cool things you can build. I'm going to show you
briefly this one that I mentioned here. There's a ton
of them here in Hopsterick's tutorials. The one I'm going to show you briefly
here is called credit scores. Now, I've actually opened
up the notebooks already, so this is just running on my laptop here. This is
the hopster tutorials that I've checked out locally.
Do I need quick start? No, I don't. Let's move to the first
one. So in this case we've got the advanced tutorials,
credit scores. So what this is going to do is I actually have two feature
pipelines at the beginning. Sometimes you'll have two, one for historical data to create your
feature groups and add all the metadata. And this is a backfill
one. So what this one is doing is basically reading some csv files
with our loan applications, bureau balance,
credit card data, payment installments and a lot of other data,
doing some feature engineering. It's also
showing you doing some EDA as well, which you typically would do,
and then creating the feature groups.
This is where I ran it, I think, earlier on. So there might be some
diagrams in here. You can see we're trying to understand basically
the data before we get into the process of modeling.
But obviously the kind of things you do here is clean up the data,
extract any features that you need to extract or create.
And then what we can have is a feature pipeline. So this is a much
simpler program. It's going to read the data,
it's going to read the new data. And this should run on a
cadence. Now, it's a notebook, so it's really just for learning. But it'll connect to
the feature store. We can see here that it's getting a reference to
the feature groups. And in this case, we're not reading new data
from the Internet because this is credit card loans and so
on. That information is not available. So what we're doing is generating data.
So this is a really good way of doing synthetic feature pipelines
if you can't access the actual data. So we're just generating some
random data in this case, or not random, but based on the distribution of
the historical data. So it's generating new feature values
based on those distributions. And then we're just going to write to
the feature story. So you're just basically going to call insert on these data frames.
And now you'll have some feature groups in here. So let's have a look at
what the feature groups look like. We can see here,
this is hopsearch here, and we've got a bunch of projects. And this is app
hopsearch AI. So it's free to create an account. It's time unlimited.
You can build systems on it, and you can see there's a bunch of these
feature groups that have been created. One of them is
called application. We can see all the features that are in here.
There's quite a lot of features, 72 in total.
We can see how it's being used as a feature view created from it.
You can have expectations, if you want to use great expectations to do data validation
before you write to the feature store. And you can see the results of those
expectations here, the same, you can create alerts if bad
data is being ingested. We can preview some of the data that's
in there to just do some EDA. You can see, have a look at some
of the data, quite a lot in there. So just give me a random sample
of some rows there. We can pre compute statistics
over the data so you can understand the distribution. In this case, we got descriptive
statistics of the features, and we can see what's happened in this feature group
over time. So we had some data written to it earlier,
and then we wrote some more data to it later on.
So the feature view you can create is the selection of features.
So this is what it looks like here, and I'll show you the notebook in
a second. We can see the provenance of those features.
This one came from this particular feature group bureau. Another one came from
application. This is the target column that our supervised machine learning
model is going to use as the label. And then
we can see any training data sets created from it. So I only created one
training data set from it, but let's look at the
code for that. So in our feature view, what we do is we just get
a reference to these feature groups first, and then we need to select
our features. And that's what's happening in here. It calls a query preparation, but really
what it's doing is feature selection. So let's call it that,
it's feature selection. We're selecting features
from a bunch of different feature groups or tables, and we
join them together. You can add filters and things in here as well.
Now, the point in time correct join is very complex, as you see, but the
only code we had to write was this relatively straightforward
code in Python. What we can do then is
we can add transformation functions. And here what we
do is we can see, we're saying, okay,
in this particular case we're looking at a mapping
transformer here, a label encoder and a standard encoder and we're
basically applying those transformation functions then to those select
set of features that are defined in here.
So basically what it's saying is all the columns that are categorical we're
going to apply the label encoder to and all the columns that are numerical we're
going to apply the standard encoder to. And that's
all going to be captured in this mapping transformer object.
So the feature view is created with this selection of features called FG Query.
We're applying these transformations to them. We've identified the target column,
which is called target funnily enough, and then we've given
a name and version so the feature views and the feature groups can both
be versioned to enable you to apply ML Ops best principles.
Of course, once we have a reference to it and here we're just getting a
reference to the feature group again, we don't need to January. It's just if
I start my notebook from here it's nice and handy, then what
I can do is I can create training data here. I didn't specify the
version train data so it just defaulted to version one.
And I've created the training data as files and it's
a random split with 20% test data and 80% training
data. Now the train data
sets being created so I could have kept in the same notebook and just
gone ahead and read the train data and train. My model I separated into
two notebooks. So these two notebooks really make up the training pipeline.
In a production system you typically have a single pipeline.
But what we can do here is we can get back our feature view and
say, hey, that train data has been created as files. I just want to read
it up. And I want to read it up, split into my features,
the training and test set and then my labels, the Y train and
y test from the training and test set. So that's great.
Now I can do some messing with the data here,
just cleaning up a little bit. Then I can use just a simple model so
I can use scikitlearn random forest classifier, I can use
XgBoost. I'm just going to fit to my training
features, sorry, my features in the
training set and my labels in the training set. And now I've got my model
and I can predict on the test set and check the accuracy of the model
and save the model to the model registry. Then I have an inference
pipeline, in this case the batch prediction pipeline. It's going to go
to the feature store, connect to the model registry
as well, download our model, get some batch data that's
going to score with in this case it's getting all the data. But we can
specify a time range or a set of ids that we'd like to score for,
and then we can go ahead and make our predictions,
and then we can save those predictions. So what we're doing is downloading the model
first. But once we have the model, then we can make predictions on our
data frame and we can save those then predictions to
a data store and we're done. So that's it. That's the kind of end to
end machine learning system that you can have and build with
this feature training inference pipeline
architecture. And we do the feature store model registry as the data layer,
pulling it all together. If you're curious to find out more,
I work with Hopsirix, which is a core part of this new serverless
machine training stack. You can go to app hopsearch AI and create
a free account or join our slack. But if you really want to learn more
about building these service machine learning systems, I recommend you take
a course that I developed called serverless ML. It's all Python,
pure Python. The course came out in the fall last year,
so we weren't using modal or hugging face. There is no hugging face there,
I should admit, but it uses GitHub actions instead of modal.
But they're both great tools for orchestrating and running Python
programs. If you want to learn more, go to serverlessml.org,
give it a whirl, and good luck on your journey on serverless machine learning.