Implementing AI and data: how to get started

You surely know by now that AI and Data have many practical applications to improve your business.  Good. Becoming aware is the first step. But then, you need to get your hands dirty on it, and… IMPLEMENT it.

Here’s where reality comes… and so few talk about it.

At Altitude Software, in addition to delivering Data-based AI to our customers, we also implement our data-based algorithms in order to improve our own efficiency.

I’ve been leading this Digital Transformation journey for the past 2 years, and I therefore know very well that, on top of the best possible algorithm, as Altitude provides, there are other key factors for your success in improving your business through implementing AI and data.

Here you find some key learnings and thoughts that I would like to share with you today:

Big Data vs Small Data/Smart usage of the available data

Machine learning in general gives better results with big data, allowing for faster and more efficient ways of designing algorithms (example cat/dog classification).

And a lot of data is already available, any company has it. However, sometimes there is “not enough” data, not everything is BIG data.

Does it mean I cannot use the data? Does it mean there are no patterns to be found because I do not have enough data?

It depends. Not all problems need much data to be solved. At worst, if you do not have “enough” data, it may mean I may not be able to have an algorithm which learns by itself from data, but you probably have enough data to automatize many answers to certain behaviors and events. We can create a guided learning.

Conclusion: start with your available data, even if it seems not much.

One step at a time

Once you start small with implementing AI and data, there are some steps which will eventually take you to a full use of your data:

1. Obtain/create your data if you do not have it

There are many data around us, but not all of it is accessible or stored. There are privacy and confidentiality aspects that need to be respected, through specific agreements or eventually anonymization.

2 . Do “the introspection exercise”: know yourself, know your data, enrich your data, profile your problem

You need to organize/structure/classify your data. This is where you will probably spend most of the time! The system will eventually learn to do it by itself, but to begin with, you need at least to decide what the classification and processing should be (and probably do it manually at least the first time).

Of course, the process will bring surprises: you thought of an initial classification and in the process of classifying you realize you are missing this or that aspect. You learn this by analyzing your data, and it will take you several iterations until you feel confident that you have a good classification model.

This first pre-classified data-set will already allow you to create a first Knowledge Base for a manually guided learning. Good visualization tools can also help you get a lot of business value just from this first step.

3. Process your data

Prepare it for the algorithm. If this is centrally done, this may be pretty straight forward (at first), although some iterations are also expected until you find the right processing.

If this is edge processing (in the source like in IoT), then you will have to think carefully of the type of processing you want to do, as changing it later in the source may be cumbersome.

Once you have a first dataset, processed and organized, you can then move into choosing a ML algorithm and train it, evaluate and tune it. This may take several iterations between steps 1, 2 and 3 until you get satisfied with the results.

4 . Choose a ML algorithm

Train it, evaluate and tune it. This may take several iterations between steps 2, 3 and 4 until you get satisfied with the results.

5.  Here is where you start getting traction

You started by manually classifying your data and it was huge work. But once the algorithm has learnt to classify with confidence, you will then be able to move into automatic pre-classification of data (with some confidence), predictive modelling and optimization algorithms. The algorithm will eventually be able to complete your KB and suggest new data which may be missing in your data set.

Beware of the pitfalls in your Data journey

When implementing AI and data, there is no “universal AI algorithm”, it all depends on your data, and it will have to be adapted to your data. So, do not forget there are some pitfalls to avoid in the data journey:

1. The first challenge: get good data to train your ML model. It seems obvious, but:

• “Trash in, trash out”: If the initial data is poor quality (a bad organization of your data for instance), you get poor quality results.
• Sometimes we forget that the algorithm will only predict or represent the universe of your training data. If my data only contains Spanish expressions from Spanish people, for instance, it may have trouble making suggestions for Latin-American Spanish conversations. Make sure you have data sets representing all the different universes you want your algorithm o learn about.
• Do not forget to constantly update your training datasets, so that you do not lose touch with reality.

2 . Where to process the data: central processing vs edge processing?

If you go for the latter, and sometimes you have no choice, you need to be sure of what the right processing should be, because changing it when it is distributed in many devices may not be that easy.

3.  And finally, Change Management:

when you introduce automatization through the use of data and ML, of course it will imply changing many processes in your company. Even for a technological company like Altitude Software, where employees are technically savvy and ready to adopt the latest advances, this represents a challenge that cannot be minimized. You need to work the human factor of change adoption in order to guarantee success.

In the end, implementing AI and data successfully is about 3 key success factors:

• Vision: be ambitious, know where you want to get at.
• Technical knowledge: this is about applying technology to solve a business problem. Understanding the available tools and possible issues is fundamental.
• Leadership, results orientation, flexibility and persistence. There is no magic, it is all about the combination of business, technology and people.

I’d love to hear your insights on this subject!

Drop me a message

and we’ll keep the discussion going!