Re: Containerized ML workflows on Fedora

On 8/26/19 9:19 PM, Michael McCune wrote: > On Mon, Aug 26, 2019 at 2:35 PM Benson Muite <benson_muite(a)emailplus.org&gt; wrote: >> >> On 8/26/19 8:42 PM, Michael McCune wrote: >>> On Mon, Aug 26, 2019 at 3:10 AM Benson Muite < benson_muite(a)emailplus.org&gt; wrote: >>>> On 8/22/19 3:57 PM, Michael McCune wrote: >>>>> On Thu, Aug 22, 2019 at 3:37 AM Lumir Balhar <lbalhar(a)redhat.com&gt; wrote: >>>>>> Hello. >>>>>> >>>>>> On 8/21/19 3:37 PM, Michael McCune wrote: >>>>>>> hi all, >>>>>>> >>>>>>> i am writing today to get a sense for how the group feels about >>>>>>> creating content that focuses on containerized machine learning >>>>>>> workflows on fedora. >>>>>>> >>>>>>> i have been creating cloud native ml applications for a few years now >>>>>>> and most of my work begins on fedora inside of containers. from the >>>>>>> wiki page[0], it seems apparent that one of the major focuses of this >>>>>>> group is around packaging various frameworks and tools for fedora, but >>>>>>> i am curious how the sig feels about embracing a more container >>>>>>> focused approach to machine learning development on fedora. >>>>>>> >>>>>>> i think the initial outputs from a push like this could easily see us >>>>>>> creating some tutorial content and fedora based images for doing work >>>>>>> that uses modern machine learning frameworks (eg spark, tensorflow, >>>>>>> etc). i also think that focusing on containers gives us an easier way >>>>>>> to tell stories about machine learning on silverblue and fedora >>>>>>> coreos. >>>>>>> >>>>>>> with that said, what does the group think about this? >>>>>> I think that this is a great idea. >>>>>> >>>>>> One of our goals is to gather success stories about how the Fedora is >>>>>> used in AI/ML industry and write some blog posts about it with a "hello >>>>>> world" examples using the tools already available in Fedora. >>>>>> >>>>> so, i guess what i am proposing are stories about how a fedora user >>>>> can unlock new power through the use of podman and buildah ;) >>>> This seems nice >>>>>> If we identify that some of the working pipelines make sense also as a >>>>>> containerized Fedora instance we can definitely create and maintain a >>>>>> ready-to-use container and use that also in blog posts mentioned before >>>>>> to simplify things and/or offer a different approach how to get it working. >>>>>> >>>>> a "ready-to-use" container sounds like an interesting idea, i'd love >>>>> to hear more thoughts about that. >>>> Would machine translation pipelines also be worth including? Fedora >>>> Translations are expected to move to use Weblate ( https://weblate.org/) >>>> which can be integrated with various machine translation engines, >>>> http://opennmt.net/, https://github.com/facebookresearch/UnsupervisedMT, >>>> https://github.com/moses-smt/mosesdecoder, http://thumt.thunlp.org/, >>>> https://marian-nmt.github.io/ >>> i think this definitely sounds like an interesting use case. >>> >>> imo, we could easily show off several different types of workflows and >>> uses that all utilize fedora and the container ecosystem for the work. >>> i think it would be great to have some articles about each of these >>> different techniques, with examples that a user could try out. i'm >>> just not sure where we would put these articles and if the ml-sig is >>> the proper organization to drive them forward, but it seems like >>> something in our purview. fair warning, i am new to the group here =) >> Ok, this may also be of interest: >> >> https://www.paddlepaddle.org.cn/documentation/docs/en/1.4/beginners_guide... >> >> Note that documentation on installation in Mandarin using Docker is a >> little bit more comprehensive than in English: >> >> http://en.paddlepaddle.org/documentation/docs/zh/1.5/beginners_guide/inst... >> > when we start talking about containers in fedora and workflows, i > think we should be looking for opportunities to show how podman and > buildah work and how they make life better on fedora. I think that we (ml-sig) should start with something simple. I can imagine an article about some well-known AI/ML hello-world problem to show that in Fedora we have RPMs ready for AI/ML engineers. It might happen that we also identify that we are not ready for some centrain worlflows so we can fix it right away. Then, when we identify something more complex - for example with Tensorflow which cannot be installed from RPM - we can write a longer article where we show how to install Tensorflow from wheels and how to use it (again some hellow world example is enough). And in an more complex example like this, we can show how to use buildah/podman and/or we can create an image for a workflow described in an article if it is common enought that it'd make sense to maintain that image.

cover in articles. Also, I think that the best platform for this is https://fedoramagazine.org/ > > no slam on docker, but these are the type of instructions we could > make more fedora-ish by using the native open source tools. thanks for > the links! > > peace o/ > >>>>>> We (Python maintenance team) already did a similar thing with >>>>>> fedora-python-tox: https://github.com/fedora-python/fedora-python-tox >>>>>> >>>>> i will check this out >>>>> >>>>>> Do you know about some good examples we can start with? >>>>>> >>>>> indeed, i do. i contribute to a community (radanalytics.io) that >>>>> focuses on machine learning on openshift. we containerize tools like >>>>> apache spark, and then demonstrate how to create application >>>>> pipelines. much of the spark work i do begins on fedora using these >>>>> containers from the community. although spark is not packaged for >>>>> fedora (for good reason), we can still utilize it by running the >>>>> community images (centos based). >>>>> >>>>> if this style of workflow is useful to the fedora community, i can >>>>> definitely write some fedora-specific blogs about how users can run >>>>> spark-based machine learning workflows. i guess my larger question is, >>>>> is it appropriate for us to demonstrate workflows that use tools which >>>>> aren't packaged for fedora? >>>>> >>>>> peace o/ >>>>> >>>>>> Lumír >>>>>>> peace o/ >>>>>>> >>>>>>> [0] https://fedoraproject.org/wiki/SIGs/ML >>>>>>> _______________________________________________ >>>>>>> ml mailing list -- ml(a)lists.fedoraproject.org >>>>>>> To unsubscribe send an email to ml-leave(a)lists.fedoraproject.org >>>>>>> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ >>>>>>> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines >>>>>>> List Archives: https://lists.fedoraproject.org/archives/list/ml@lists.fedoraproject.org >>>>>> _______________________________________________ >>>>>> ml mailing list -- ml(a)lists.fedoraproject.org >>>>>> To unsubscribe send an email to ml-leave(a)lists.fedoraproject.org >>>>>> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ >>>>>> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines >>>>>> List Archives: https://lists.fedoraproject.org/archives/list/ml@lists.fedoraproject.org >>>>> _______________________________________________ >>>>> ml mailing list -- ml(a)lists.fedoraproject.org >>>>> To unsubscribe send an email to ml-leave(a)lists.fedoraproject.org >>>>> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ >>>>> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines >>>>> List Archives: https://lists.fedoraproject.org/archives/list/ml@lists.fedoraproject.org >>>> _______________________________________________ >>>> ml mailing list -- ml(a)lists.fedoraproject.org >>>> To unsubscribe send an email to ml-leave(a)lists.fedoraproject.org >>>> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ >>>> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines >>>> List Archives: https://lists.fedoraproject.org/archives/list/ml@lists.fedoraproject.org >>> _______________________________________________ >>> ml mailing list -- ml(a)lists.fedoraproject.org >>> To unsubscribe send an email to ml-leave(a)lists.fedoraproject.org >>> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ >>> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines >>> List Archives: https://lists.fedoraproject.org/archives/list/ml@lists.fedoraproject.org >> _______________________________________________ >> ml mailing list -- ml(a)lists.fedoraproject.org >> To unsubscribe send an email to ml-leave(a)lists.fedoraproject.org >> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ >> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines >> List Archives: https://lists.fedoraproject.org/archives/list/ml@lists.fedoraproject.org > _______________________________________________ > ml mailing list -- ml(a)lists.fedoraproject.org > To unsubscribe send an email to ml-leave(a)lists.fedoraproject.org > Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: https://lists.fedoraproject.org/archives/list/ml@lists.fedoraproject.org _______________________________________________ ml mailing list -- ml(a)lists.fedoraproject.org To unsubscribe send an email to ml-leave(a)lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/ml@lists.fedoraproject.org