From Apple Engineer to AI Pioneer: My Unexpected Journey (Podcast)

People are , wow, this is magically. , yeah, it only took me 15 minutes, or it took me one promise. Welcome to episode 14 of tool use. This week, we're going to be covering 10 different tools that you should be leveraging in your flow in order to maximize your use of AI. If you haven't joined Tool Use before, it's a weekly conversation about helpful AI tools where we teach you what is possible today with real examples. Today, we're joined by Ray Fernando, a former Apple engineer turned prolific builder and live streamer teaching the whole world how to code with AI. Ray, we're super happy to have here today. Welcome, to Tool Use. How you doing? I'm so excited to be here. This is really awesome, and I'm excited to share a bunch of knowledge for your listeners. I got a lot of cool stuff in the books and even some other stuff that I've been cooking with that you guys would be so interested to check out. I'm super excited. Well, how did you get into AI? Can you tell us a little bit of your story? Yeah, the story is a little funny because it's totally non-conventional, and I feel that's the story of my life. I used to be an engineer at Apple for 12 years, and then I came down actually with long COVID. So I got COVID in March 2020, and then I was so sick that I really couldn't focus. I had brain inflammation. I really couldn't move. It was really kind of crazy. So I was going back and forth, and I had to take some time off. And then after several years, it wasn't getting any better. So that was really, really kind of scary for me. And then after that point, you know, I was basically , oh, saying, hey, let us know when you get better. And then that's kind of, you know, what happened with me? I was , oh, wow, what am I going to do with my life? and I started playing around with Tet GBT, and I thought, wow, AI has actually come along a very long way. And that's only last year that I started playing with it. And then that's kind of what really opened up my eyes as far as the potential here. I was still going through lots of brain fog at the time. So it just had very limited cycles of things that I could focus on. And I had to figure out, you know, how can I use these types of systems to, you know, leverage my knowledge and other things I wanted to do. And that's kind of really how I started to kind of get into playing around. with this AI stuff. As my health progressed in the following year, I started to see how much more intelligence, you know, with GPT4, and then also with the Claude stuff that was coming out, it said, , wow, you can actually leverage these language models to do so much more for you that you can really realize. And I don't think people really understand the potential that's hidden behind these machines or the discovery that these people have made. I think people in our industry totally understand, but the broader audience of the world doesn't really understand how much is going to impact all of their lives in every single industry. So it's going to be computing, it's going to be cooking, it's going to be medicine, it's going to be every single thing you could ever dream of. And it's pretty obvious to us, but it's through this discovery process of playing with this models that I start to kind of find out about its power. And figuring out, especially covering an engineering background, I want to figure out, you know, what are the inputs, what are the outputs? It's, you know, just letting my curiosity went wild. So that was kind of how I got into the AI stuff and got into prompting and so forth. So that's a little bit of how I got started. Really, it's kind of a weird story. But I also going back even further before Apple, you know, I basically dropped out of school, right? And I was just hustling along, working any type of gig. And literally it was, you know, trying to do everything from wedding photography to DJing to , you know, pretending to be on the radio station and being a student and stuff that. all this crazy stuff in the world just to get my feet started and get off the ground and, you know, got all these lucky breaks and just kind of a door opens up and I just, you know, run right to it because it's the right time. Very cool. I love the story. When you started building with chat jeepa and other projects, were you doing something just for fun, were you trying to solve your own problem? Was it even just a contract for someone? How did you get into actually building with these tools? It was literally just helping me, right? Because I have severe dyslexia. So that's another problem. It actually started to come out. a lot more through the brain fog that I didn't really realize I had. You know, I had worked around it through audio. And all the techniques I was using for memorization were based off the video and audio. So whenever I would see something, it would instantly stick in my mind. I can walk into a room and memorize everyone's names. I can redraw stuff, you know, just by looking at them. I can tell you everyone what happened six months ago who was standing where and all these details. As you can imagine, it's immensely helpful if you're working on these really complicated build systems inside of a proprietary large closed source or OS. And so it was just really easy to remember, oh, that change happened six months ago, and that kind of led to this. And so that was kind of the beginning. It was , I lost my ability to do that. And that was really scary. And I felt that my whole entire identity for how I do everything was gone. And I thought, how am I going to operate in life? This is a huge handicap for me. And I don't know what I'm going to do. And so I reached out to chat GPT to try to help me with that. It's , okay, can you help me read these things? Can you just distill this down to something simpler? I can't focus for longer than a couple minutes right now. It's just so hard or it takes that much more energy. And that was kind of the beginning of the power of this type of systems. And then I said, can you help me code? And that was a really crazy thing. It was I started writing more apps in Python and started writing to databases and started to do any things. It's , this is really good. And this is only GPT 3.5. But hey, this is amazing. And then the 4-0 came out, and that was just an even bigger bomb. It's , you're kidding me, that this can actually do this, authentication. I could translate to a different programming language if I wanted to. And so then that really opened up my mind and said, well, I've always worked on client-side software. Why can't I work on more web frameworks? I really want to extend all the things that I'm doing in Python into these other web frameworks, you know, and that's kind of where this whole , you know, deep sea of abyss of, you know, the language model is kind of actually having more intelligence has really helped me reach out and translate my ideas in my head out into these apps and things. Interesting. I want to circle back on something you said earlier about how, , we can recognize in our industry how important these machines are going to be for every single aspect of our life. And it's crazy. When I talk to, , friends that I have that aren't in this space, you know, I'll be , well, AI's doing this and it's doing that. And they're , you know, they'll nod. And they'd be , oh, it's crazy. Do you catch the game last night? I want a chance. It's they just don't even care. it's they're or they'll do something we're , hey, you know, I talked to chat GPT once and it was terrible. That's right. What did you ask it? And it's , why I just asked it how it was today? Or it was some super simple proff. And , of course it's not going to seem good if you don't know the right things to ask it. So it's it's been really tough for me to get my friends and family in on AI because it's a hard conversation to have. So you're , well, what do I get it to do for me? , well, what do you need from it? Do you try to get your friends and family to use AIs or anything you tell them to get the normies in on AI? I agree. No, I do the same exact thing and they just see me so enthusiastic, just incredibly. I was , Ray, this must be serious because I don't normally see Ray just, you know, off the wall is bouncing around about something, you know, that's really interesting. So they use it and then they come back to me , hey, I try to ask you, you know, chat GPT to do this. , can you help me? And, you know, just my natural genius bar, you know, geek squad, you know, having worked type of thing, you know, just kicks right back in from decades ago. It's , oh, yeah, this is what you do. But this is where I start to see this big dissonance or just this large gap. And it kind of brings me back to my early days of Apple. It's they cared so much about user experience. They cared so much about being a user and trying to solve those problems. And so what most people don't realize because it's all closed source, underneath the hood is just that deep. iceberg of decades and decades of code and stuff that comes together. I mean, all the way going back from the next step, you know, that stuff is in there. all that stuff is in there. It's not a secret, right? It's literally that is all running and somehow it all turns on. And there's just this magical experience that is really driven from that level on up. So that's actually kind of what I'm seeing right now. We have this unique ability with the way that AI is kind of booming. And I kind of want to remind people that we're still so early on this. And this is why I'm really excited about is that it's sort of our job right now as engineers to build these app experiences, to think about things as a user, just really sit and observe and ask questions and understand what people are kind of missing. And to try to bridge that gap and try to figure out, are there frameworks that can help me get there? You know, if your specialty is web, you know, what are you doing in that space that can help you bridge that gap so that you can bring a cool experience in. If you're working on client side code, you know, what are the cool things or cool tricks that you can do, you know, how you, if you're really good at loading data, can you partner up with someone who is really good at UX? You know, if you come together and then you guys are both leveraging some of these AI capabilities, you can provide an experience that is so much more rich or take over an entire industry, literally just with a couple of people. And sort of that's an interesting example to how I've started getting into my transcription stuff. I had no clue that people were paying up to $7 a minute to have their audio transcribed when that can literally be done in minutes, you know, with these new language models. And it's very obvious to us, but there are, the app experience is totally not there completely, right? People are trying these different things out. And I feel there's a large opportunity, even just as that one example, but in many of these industries, or it seems obvious to us you're saying. But there's a, you know, people are trying things and they kind of get stuck. And that's where you have to go in as an engineer, kind of, you know, take off your engineering hat and then put on your learning hat and watch these people use their software and see where they're struggling. Because once you solve those problems, it becomes sort of magical to them, right? And that's hours of engineering and that's what you do really well that you can solve. But it's , don't come up with the solutions first. Just observe and see what people do. try to make it as simplest possible for them. Oh, great advice. Love Living by that. As we're about to release the Open Interpreter desktop app, we've been doing a lot of the same, work with people figuring out how to use it, how they're using it, what the best ways to approach it. But in regards to helping bridge the gap and understanding, would you mind sharing some of your workflows with us? Yeah, sure. I'd love to share some workflows. I'm really excited because I have a couple of things I've been working on, and I think that some stuff would be really, really helpful. And I think the first workflow that I use a lot, especially as a live streamer, so literally just transcribe audio. There's the first problem I had, and I thought, hey, it'd be a really good idea if I can transcribe my live screen because there's so much knowledge in there. I had a three-hour live stream. 10,000-plus people saw it in the first couple of hours, you know, for using Claude Sonnet and, you know, coding with cursor. And there's just still many people watching it today get so much value. And it's , you know, no one's going to watch three hours unless there's a huge burning desire to watch this type of thing. And if I could just share these snippets out, it would be super helpful. And how do I do that? I have to get the transcription out. I built an open source version of a transcriber app called the MLX auto subtitled video generator. I literally just forked this from an existing creator. And I wanted to leverage the power of these Apple computers to run these language models. And, you know, this is something that just needed a very basic user experience. but the goal is just to drag your video in, and what you can do is just once it puts the video in, it'll just do the transcription using a whisper language model. And depending on the power of your machine, this can take somewhere in the minutes rain to tens of minutes to do the transcription. And then what it does is it generates basically a text file or SRT file. And, you know, this was obviously written with assistance of AI code editors and so forth. But this is just one of the tools that I started to use. to make this thing. And what's great about it is open source. It's actually featured by Cocktail Peanut, which is a one-click button downloader app thing, which you can use to actually, you know, run this thing. So I have some notes here on how you would actually install this. And so you obviously need a Python environment, Konda environment, and so forth. And, you know, it's quite a bit to get set up if you're not familiar with writing code. And even if you do write code, it's still kind of a headache to sometimes, you know, make sure your environments are correctly. and I actually ran into a problem. I made a little guide here that actually sketchy set up running Python so that you can actually run it on an M1 Mac. I accidentally installed the Intel version of the Konda, and so that really messed up all my dependencies, and I had to remove it, and then reinstall the M1 version, so I actually wrote a full guide on my blog at rayferndo.a.i, and that's also linked in here. But yeah, basically, I also, given the demand, I actually wrote a web version of this where you can actually transcribe the video. So I could show you the difference right now. So if I get a video, I'll run the current Python version so you can see the speed differences. So if we do browse files, and I'm going to get this one here with a video interview with Chris Latner. It's about 49 minutes long or so. You can upload this. So this is using the MLX video transcriber. And so this will just use the power of my local machine to do the transcription. And so as we see it going on here, you'll see it kind of running. It downloads the model if it's not downloaded yet, and then it'll just go ahead and start fetching the files and actually do the transcription. But the one I wrote for the web is actually a lot faster, because as you see, this year, it's going to take quite a while to hit that 49-minute mark. I could just select the video, the same video here, and just have it done completely. It doesn't matter if you have a Mac or it doesn't matter if you have a PC. I'm making this app called raytranscribes.com, And this is just going to be done right over the web. And so it'll just process the audio on the machine and then generate basically the same transcript for you. And so this is something that I developed. You know, I wanted to take my Python and extend this out into a web framework and stuff. So I started to, you know, learn NextJS. And in that process, I thought, hey, wow, there's a, you know, it's a really rich framework that has an AI SDK. And here it is all transcribed. And if we check the, the one that's being done on my Mac, it's still at the 20 So we're only halfway through the one that's running on my machine locally and my web version is already done. And we already kicked that off quite a bit later. So I'm going to quit this so it just isn't eat up all my machine machine resources With this text file or this the SRT file now you actually have these rich you know documents that you can use to start doing some cool stuff So this is actually you know everything transcribed very accurately And this is something that I needed. And so in my workflows, , this is something that I really use a lot. And I'm actually planning to launch this app as soon as probably when this podcast is coming out or when this podcast would be live. This is actually, you know, my app will be live. So people who are creators can use this. If you have a Mac, if you have a PC, you don't have to install any Python stuff. It's literally on the web. It'll be just as fast as this. And I want to make it accessible to everyone so that they can use it. And so I think the next step here is said, , now that you have this transcript, what do you do with it and how do you get some information out of it? And so what I usually do is I basically just copy my transcripts. And I go over into Cloud. And I have these projects here to generate blog posts and do different things here. And so I have one that basically says generates a blog post from my transcripts. So I made a second version here. And so with Claude, you can actually set up these projects and give it instructions. And what you can do is actually have it do some analysis beforehand and then have it generate the blog post. So all I do is just copy and paste that text into here and hit run. And this is literally having your own app for you. And I would encourage people to use cloud projects a lot. I know sometimes you want to jump into code right away. Sometimes I do too. But I feel this is a really fast testing ground to test my ideas and test prompts out and iterate. And so right now what it did is just generate this artifact. And this 49 minute interview with Chris Latner is now, you know, has a complete blockpost that I could just publish right here. Just as soon as the live sheet done, there's interviews done, you just dump the whole audio in there. and you have the entire analysis of the whole thing. And it actually gives you title recommendations and it gives you the AI analysis of the main topics and key points. And also it even tells you the target audience. So if you're actually writing this stuff, you can actually figure out, okay, this is a, you know, geared towards AI engineers or developers. That's really important to know. Also the emotional elements. Who are you actually targeting and what type of messaging are you going to do as far as title recommendations as well? And so a lot of these types of prompting stuff were inspired by some of the stuff I learned online through Twitter with people actually doing these major discoveries for prompting. And so I've been kind of diving deep into prompting and I could probably share. I'd love to share this prompt with your listeners. You know, it's just as a bonus business for tuning in because this kind of give you an example of how you can structure prompts and get some really interesting insights from it. And this is just probably one example of a workflow that you can design for people. And how we were talking about earlier for people who have, you know, I tried to use AI, tell it to write my blog post. It's not going to write a detailed blockpost unless you give it some detail information. So that's where we kind of start out with something this, give it a transcript. And if you then just tell it to write a blog post, it'll still be okay. But with the insight here and the leverage that we're leveraging is the instructions to do an analysis beforehand, to get the inputs. and then to find out the target audience. And so by telling the language model, these additional instructions, and to kind of get it to focus and kind of say, okay, I want you to find the target audience, I want you to find the emotional elements, I want you to find any constraints, any relevant keywords and things that, 46 main themes, and then write a blog post, you're going to get a very rich document in the way that we did here. And this is just, you know, one of many interesting techniques that I feel , you can leverage in these language models to just take your prompts and just make them 10x better. And then what you can do from here is build the user experience on top of that. Imagine if I just release a blog post generator that just literally takes your transcript and just generate blog posts. That'd be cool. I might just throw up a landing page and see if anyone's interested, but that would be really interesting. Yeah, because this is the blog post. It writes everything about why we need a new programming language. So this is exactly what Chris Latner was talking about. He was explaining all these different problems. And Mojo is, you know, his platform there. And it actually kind of breaks down everything there with the key technical innovations as well. I mean, that's amazing. That's, to me, I'd be happy to post this stuff, you know, with some review. And it gets you kind of started there, you know, for getting your content to be out there. The other problem I realize is that once you have this audio podcast, Google is not really going to find it online unless you post about it or write a blog post about it. So the impetus here is to take this really rich content and produce things so that Google can start seeing, oh, Mojo, people are searching Mojo right now because Chris Latner was on this podcast with Joe Rogan or something. And so then you want your blog post to show up above everyone else's because you know, you want yours to be out there. And that's, you know, the SEO game you could say. But you do have rich stuff. You know, this is a rich interview that was done at the AI engineer conference. and no one can discover unless I post this. And, you know, I want people to discover it. You know, I took this time to go record it. So I should, you know, post this. So I feel that's kind of one of the ways that you can work. And so I thought it'd be interesting to also maybe kind of break into this new segment here with , how do you start with a basic prompt? And then how do you make it so advanced that? Or how do you add more rich or more depthness to it so that you can get the language model to give you better? better output than someone else was just saying, you know, do a one sentence task. And so the way that you usually do that is just for me, I just start with a workflow. I literally will just go into Claude and I just gave it the SRT file. The entire SFT file was , here's a bunch of data. You know, it's not cleaned up or whatever. This is just, you know, I'm planning to drop this file in. And then I literally just gave it that one line. It said generate YouTube timestamps. And the timestamps will go in the description of the videos so the users can click on the timestamp. And so this is probably what most people will see when they tell that to do that. It may not even be that accurate. It's just going to be , here's, you know, here's what I was thinking for this. And so then it's always good to continue to go back and forth with your prompts and stuff because the goal here is to build up this, this basically this history of conversations so then you can get some instructions out of the language model later. So as we see here, the stream is about three. three hours and you know I'm just giving the language model feedback saying so I want the time steps to go from beginning the end you're off to a good start so just kind of giving it an additional encouragement there just in case the AI wants to come and destroy me. I was oh no Ray's really cool I'm gonna go for someone else because this guy was really nice. So folks it's really important to be nice to your AI as we see there the the timestamps you know so I just kind of giving it more iteration here and I think the the key thing that you should consider it's once you have kind of what you're looking something that's really close to what you want you want to take this and now you basically have created the machine that's going to stamp out everything else for the rest of your life right so it's you know doing a process by hand you the process you know how does macdonalds make these burgers so consistently all over the world is that they've nailed down the process and now they can just copy it at every single location and everyone follows those exact same procedures over and over again and so you get some repeatability and you also have some variability with water and various things but it's still fairly down to a science and I think that's kind of the way I want to think about these things as an engineer so the technique that I use here that you want to use if you're paying attention folks right now is describe the technique you use to get meaningful time stamps out of such a long transcript so that other LLMs you can reproduce the results please give the instructions in Markdown, here are the timestamps you generated that were useful. So I'm giving it some feedback about timestamps that were useful, but also telling it to give it instructions in Markdown. And so there's something very special about Markdown that I've sort of discovered about a year ago when I was copying stuff from Notion over into ChatGPT. And what happens is that whenever instructions are in Markdown, the language model seems to follow them almost exact with procedures. And there's another pro tip is that if there's actually numbers, it will actually use the numbers as a ranking system to saying number one is the highest priority. And if you have number seven or something, it may not get to it. It'll prioritize number one. If you do markdown with a list format, it will try to follow the lists, you know, in line by line. So there is actual hierarchy differences and you want to play around with it just to see what can happen if it's not following instructions because you just give it too many numbers. So just so you're aware of that. But what I find is if you just tell Klaude to generate the markdown, if you have this just general idea and you just have a conversation with it in a chat and then say, hey, I try to do it and here's my prompts and please improve it. I'm not seeing number seven execute. Please improve this and generate the response in markdown. It's going to give you those instructions in markdown in a better format because it apparently is really good at giving itself instructions. And so now we have this new instruction set that I can now use in Clod and generate this in the new project and literally all I have to do is just give it a markdown file. And you can open up a new chat window and paste that actual prompt in here and then continue to improve it. So the difference is when you open up a new chat window, what's nice is now you have a larger context to run. And for those who are just getting started with language models, once you kind of go very long with a chat, they also encourage you to open up a new chat window because then you can leverage more of the model's intelligence. and it kind of helps it focus. So think about your desk in some way. If you have a desk and you have a lot of paperwork on there, sometimes you just need to clear out your entire desk and start fresh again so you can get just that one idea executed. You don't want to have too much stuff on your mind. And so the language models kind of behave in that same way. You want to clear out the desk, you know, open up in your chat to clear out its mind and kind of really get focused and really get into one specific problem so that you can solve. So experiment with that as it may. And so once you have these instructions, what you can do for me is that I want you to experiment and try different language models. And one of the ones that I've been playing around with lately has been the new Gemini Flash model. And that model has been extremely good at taking these system instructions I have for generating timestamps. And it generates very, very accurate timestamps. And so what I do here is just going to ais studio.gov.com. And in those system instructions, I literally just put the entire markdown file in here. So I just put the huge SRT file in the text format, and I just tell it to, you know, rerun, basically. rerun, I think it's running now. I should be running. Yeah, so it's running at the bottom. And there it goes and generates the timestamps. And so one of the reasons why I landed with Google Gemini or why I'm using Gemini Flash, and you can try different models here, was because somehow this out of chat GPT, GPT 4-0 Mini, all the different types of models, this one had the most accurate timestamps. And every model is going to be trained differently. So that's the other thing. Google actually may have a lot of training data on YouTube and timestamps and referencing types of languages. And so that's probably its advantage that no one's talking about. But you'll discover that as you play with it. So don't get discouraged if you don't get results from one model. Try different models. Try different temperature settings. Try, you know, different types of things that can actually change the result for you. And you'll be pleasantly surprised. Even just changing something as temperature can make a huge different. in your output. And so this is basically one of the basic run-throughs that I'm using to generate app experiences. So if you want to, you know, generate an app, you know, you can, this is, this is kind of the workflow that you do it. So I actually made an app that I'm currently trying out right now. So this is I just generated this with NextJS. I went into V0 and I said, hey, let's just see if I can actually, you know, generate timestamps with this. And so, you know, , for example, I'll get that one thing that I just made here. I'll just drop this in here. So all I have to do is just drag it in here. And the language model using NextJS is just going to stream back those timestamps for me. And here we are. There's an app, right? So we went basically from prompts to an app, you know, all just through that process. And I spent a lot more time in the prompting phases because I wanted to make sure I had the correct output that I wanted. And once I was really satisfied with that, then it was just easier for me to move into the code phase to generate things. Because sometimes I could spend way too much time in the code. And this is just one of many app experiences that you can make. But just think about it this way, how helpful it could be for other people and other creators to just get transcripts that quickly, just from these, you know, types of SRT files. And so as you can kind of see, I'm building this different sets of tools to help me as a creator in my process. And it all started with really prompting, right? And prompting, I feel , is sort of the gold nugget here because you're going to leverage the intelligence of the entire world. And, you know, how do you do that? You know, you basically start out with having a conversation, you're having a conversation with an expert. Having that expert then tell itself to give it instructions so they can repeat that process over and over. Trying it out with these different models, you know, putting in the files, changing temperature settings, various things. Once you're satisfied with that, you can generate an app experience. that's going to deliver something just what you're seeing here. I'm wondering if, , some problem I have with building is, , I don't know when the right time is to jump into the code. And I think you've touched down in a little bit here, , make sure you've got the prompts, right? Make sure you've got this figured out. But then it's sometimes I'll just jump right in and start coding, but then that's too early and they get locked into an architecture that's not good for the future. Other times, recently I've had the problem of , okay, I'm going to start a project, and I'm just going to live in prompts land. I'm going to live in cloud projects. So I'm going to just dream up all these crazy different architectures and figure out the right ones. And then I end up just procrastinating too much and not coding enough. And so it kind of hard to know when you built the thing that is ready to be built in code Do you have any tips on when to know it the right time to jump into the editor That is such an important topic that you bring up because I to be honest because these are such small things I literally build when I get inspired So this UI was actually built, I think, on a live stream. I built this on a live stream because I was just itching to build. I was , God, you know, my mind was blown when I was using V-0. They changed to this new chat format. It's much easier to generate UI. And then I was so inspired. It's , what can I build in my workflow? And so I was inspired to just build the UI. So in v0.dev, I literally just said, this is my prompt. And this is kind of why it's important to play around with prompts. So I said, generate a very simple app where I can just drop the transcript SART and it'll generate 10 cents for my YouTube description. You can, you know, help. And that's all I did. literally it created this, you know? And I was , wow, it actually works. put the, you know, click here and click the SRT file and stuff. And then, you know, so here's, , the first 15 lines as far as, , a sample transcript. And then it generated, , the new version for me. And for me, I just felt so inspired to build that I felt I didn't necessarily need the code. I just needed an experience. And that's kind of what I was itching for. And you'll have these moments. And I want to deeply encourage anyone. If you're feeling in the code moment, just go code, you know. If you're feeling in a UI building moment, just go do the UI, you're going to get inspired by walking and then they'll kind of play off each other because there is this negative space that you create where you know you have to build the UI experience, but right now you're just in this moment. And as an engineer, if you're ever building and you're ever inspired, you probably get 10x to 100x more done than an engineer who's forced to code. And I think we've all been in that situation where you have a P1 or you have this high priority bug, you have to fix. You're not that excited about fixing it. You're not that inspired, but you know it has to get done because the business needs it for whatever delivery, where if you were more inspired, that same period of time, you would just have done way more in your research or output is just way higher and people are , wow, this is magical. You're , yeah, it only took me 15 minutes or it took me one problem. It's it's whatever you're feeling in the moment. And I feel that happens more often than not, especially you're already kind of thinking about these things. you also have to think about your days. , as an engineer, whenever you sleep, you know, you have stuff on your mind. That's also building up over time. Think about 70% of what you're thinking has been repeated from the previous day and the previous day before that. So a lot of these things are already floating in your mind is basically the reason why it comes to you is because it needs to get out. And you're giving it that outlet. So it's sort of a blowoff valve. So if you're inspired in the UI, just go do that. It's been in your mind for a while. It needs to get out. You know, if you're in the code moment, just get it out. You know, I just probably do more so than ever. But it also creates this other paradigm of you're just doing too much stuff and it's not really focused. So I'm still struggling with that, to be honest, especially, you know, but I'm trying to wrap things around a project saying, okay, I have a creator workflow. I want to accomplish these things, right? You know, theoretically, should I be working on timestamps if I'm not completely done with my audio transcription stuff? No, but I do, you know, I'm inspired and it helps kind of get me creative about things. As I move through this space, they're sort of still intertwined and related. But to me, I value creativity and output more than I try to figure out , okay, I need to, you know, sometimes I need to get into a forcing function. But right now there's such a boom in AI that I have to tinker, I have to play. And I feel that's how my mind works with curiosity. So I don't know if that's , I think the, I guess it's kind of summarized. Maybe if you feel inspired, get it out because something's probably in your head. it's been in your head for a little while, you just have to get it out. You mentioned, and I fully agree, the need to tinker, try different tools, try different models, make sure that you explore the landscape and see what works because we are so early. There's no real, well-defined best practices. Where do you get, you show us tools V0, Google's AI Studio. Where do you discover these tools? Where do you get your AI news and information from? Yes, a really good question. So my primary resources are X. the platform or formerly known as Twitter. I follow a lot of people on Twitter, whoever are building. If you're building stuff in AI, pretty much usually following you. And I also listen to a podcast called ThursdayI, T-H-U-R-S-D-A-I from Alex Wolkopf, a phenomenal show. It's probably the show if you're into AI that you should be watching. , seriously. This guy, the level of detail that he covers as far as models, the people who have on the show, drops for AI. Every Thursday, this guy is. just dropping so much heat. It's and he has a newsletter that keeps me up to date if I'm not really paying attention to stuff if I'm in builder mode and it's just really important. I just kind of see on the cusp oh okay there's this new model I should be paying attention to this and then another part I use just as YouTube so you know Matt Wolf is a really big YouTube person who covers a lot of these types of things. Matthew Berman is one person I've run into as well who likes to cover these models and tinker those have been the two kind of pillars on YouTube that I've been using to get information for me. But what I find really interesting is all the people who are on X who are posting, even people who have 100 followers. If I see something interesting, I'm going to follow you. And I don't care what your follower account is. I care more about creativity. And if someone's building something cool, because that kind of inspires me. And then I can kind of reach out and ask questions too. So, yeah, I definitely leverage a lot the X community because there's a lot of people in AI who do things. and really obscure people from , you know, anime characters to non-animate character people to people who are building cool stuff will only have a hundred, you know, a handful of followers. And then to that extent, I'll also say that the X spaces is an interesting environment too. With these smaller builders, they're going to be online talking about their creations and they could only be five people in that space. I'll join in and start the conversation too because people are, you know, they're sharing their same experience. and it's kind of sort of almost Discord channels in a funny way, but it's much more free. Interesting. I think one of the one of the things, reasons we wanted to start this show off too is we realize how much talent there is out there on Twitter and out there just in the ecosystem. And there's a lot of people that are under they don't usually go on the big podcast because they don't get the invite. And we found so many interesting people through this show that are building such crazy technology that have such low follower accounts you're saying. And there's such fascinating people out there. And so that's why we try to find the most interesting, prolific builders out there and get them on the show. One thing that's tough for me with Twitter is, I'm sure with all of us in the AI space, is it's just, there's just such a fire hose of new releases constantly. And it's so hard to know, , which things are worth your time because we only have so much time in a day. And we really have only so much time to try new tools out very rarely, because we're using the tools we love. Do you have, is there some sort of criteria or way that you select new tools to try? Or is it just a curiosity thing again? I'd say it's more a curiosity thing. I'm very lucky that my full-time job is to figure this stuff out, play with AI research. And so I feel every morning, every evening, I try to dedicate time to just go play. And I feel even though, I know at the back of my mind I need to shift this app or do whatever I need to do. I still need to go play and I feel to me I value that way more and it's kind of hard. I don't have a direct answer but I feel , you know, going to play and I there is a need for this. I agree with you as far as trying to reduce the noise because there can be sometimes too much noise. I have recently also taken some breaks just to get a sense of the gap of things. And what I have also discovered is that there is a bit of FOMO because a new model comes out. Everyone wants to play with it. But everyone's still back at the same problem, which is user experience. And what problem am I solving? And so, you know, is this going to be something really interesting for my use case right now or is it not? And, you know, can I leverage this or is this something I should be paying attention to? And the Gemini thing, the Gemini Flash Oshu update came out, I didn't immediately hop on it. I was just tinkering with AI Studio recently, and it's , oh, there's a new model. This is cool. Let me just try this. This is even worse. Oh, my God, it's better than my current anthropic workflow. So why not? And so, you know, that's kind of the rabbit hole kind of discovery. I'm also, just you guys, have launched my own live streams. So it's really nice that people actually are reaching out now and saying, hey, I want to talk about my stuff or I want to share this cool thing. So I kind of hear that from some friends. So I guess the moral of the story could also just be building a nice. community of friends, right? you guys are in my community now and our friends and you guys now bring me aware of things that I didn't know. It's I didn't know you guys built this cool tool to, you know, leverage and, you know, the PIP install. You can just have all these things available super fast. It's , man, that's awesome. , and so, you know, you're constantly updating it. You're talking about things. , you know, having a little , you know, Spotify playlist of podcasts that you can just kind of listen to on the walk, you know, kind of having that, that type of workflow to kind of listen to these things. But it is, it is a lot to I guess there is a big fire hose of things. But that's, I think there's, yeah, there is builder moments, but there's also moments of learning. And I feel I'm just so into this stuff and very passionate. I'm very fortunate also to be able to be at this type of position to do this. Yeah. Yeah, the passion is a must being able to have the endurance of just, you said, consuming the fire hose, filtering out all the noise. But we do you have a couple more things that we wanted to show off. So do you want to see a couple of demos from our end? Sure. Yeah. I'd love to see them. I'll show off my little demo real quick here. It's kind of a silly one, but it's something I've always wanted. So I built a little app that I haven't played a lot with visual AIs, visual image models very much. And so I wanted to play with Moon Dream because there's such a really interesting model. It's a teeny tiny little vision model, and it's very performant. And so I built a little app. I guess I can talk about the infrastructure. Actually, you know, I'm going to kick it off. Let's see here. on main dot pie. And while that kind of goes off in the background, and talk about how I actually built it, but it basically downloaded a little Moondream model, and it brings up a little dashboard. I should probably fix this little thing where it shows layout. But basically what it's doing is every five seconds, it's going to take an image from my webcam, and it's actually using the webcam over here. So the view is actually seeing me from the side, and it actually is looking at my screens, and it's looking at me. And so now it can tell right now that I'm focused, and then I have good posture, than I'm sitting in my seat. It has stats, right? So I had this thing running all day yesterday and the day before. Oh, I'm distracted. Oh, no. But I had this thing running all day, and I had a focus rate of 95%. But it was really interesting because it was fun to play with every five seconds takes a picture. It runs it through Moon Dream. And my prompts for Moon Dream is basically , here, there's an image of a desk here. I need you to tell me in two paragraphs about, , I can, give it tons of text. I want to make a bunch of texts and describe everything it's seen. It'll say what apps are on screen. It'll say if the person is in the image or not. So I have this thing called present. It's bad language for this really. But if I'm not present, then it shouldn't be counting my stats there, right? So it's kind of the military has a thing called ass and seat, you know? , my ass and seat from when I start to work, so when I'm not. So I'm not on my phone in the bathroom. Get that ass and seat, boy. Exactly. It's going to have the sound. Just, just, just click. my microphone, just do you get that. Get your ass and see, why. Okay, I think I will. But it's really funny. Yeah, tough dad. Yeah. Yeah. That's awesome. Holy smokes, man. We've talked about this on the show before about how, , I think if I was to have an AI productivity coach, I'd want it to be, , kind of mean to me because I feel I would respond better than that, than a very, an AI that's , oh, you're doing great. Try that. You know, no, I want someone to be , hey, get that. , you're saying, get your ass. and see, why aren't you staring at your screen? Get off your phone. So I have it in the problem, too, , is this person staring at their phone? This came because there was a day last week that I spent, , two hours on my phone during, , a day. And I was , that's, I can't be doing that. , I just can't be doing that. And so I was, , surely a webcam and a local model can figure this out. And so, anyway, it makes this, , two paragraphs that's exactly what I'm seeing on the screen. And then I actually use Lama 3.1, 7B, I think, or 8B I think it is. And then I figured out a way to do structured outputs with it with Lama Index. And so it takes that big wall of text. And it's , okay, from these observations you've seen, is the person focused? Are they in their chair? And do they have good posture. And that's literally all it outputs. And then I just use that for stats. And so I could do a whole lot more with this. I could even say what apps is it opening or all these different types of other things. But for right now, it's just posture and if I'm there, essentially. I think there's something cool here. I think there's even a whole company that could be built around this on a small camera that lives on your desk just to see if you're actually working. Everything running locally because you don't want this type of stuff going into the cloud. But anyway, that's what I built for this week. I love that. That is absolutely cool. I love the take also of being the hard ass. if you need some voice assets, I'd be so happy to participate. Growing up, I had a very tough father and it was just that for baseball. it was just crazy you know you could pitch the perfect game and it'd be there's always something that messed up but it pushes you a diamond right and you see these high performing players step curry who play basketball and their coaches aren't soft at them they're yelling at them they're pushing them and they know it they know it should be better and same thing oh you should make it steve jobs focused right at apple you know this guy would just yell at people right and they produce some amazing stuff so yeah i this idea this is a good is really cool And it so fun One other thing that I mention to this too one of the reasons I built this too is because I knew you were coming on the podcast this week And I seen some of your live streams about Cursor and building fast And I was , okay, you know what? I'm going to see, I have this ID in my head. How fast can I go from ID in my head to working application? And so that's what I did. I built in Python's fast, but I used Cursure Composer. I got the docs from Moondream. I just kind of pasted them in. And then I just turned on, I use Super Whisper, but I just , I voice prompting a lot, because I hate typing, and I don't think I'm as fast at typing as I am. But I just kind of turned it on, and I was , I want these kinds of things, here are my different product requirements, here's this and that. And then I took that wall of text that I made from talking to it, and I put it into Claude, and I say, can you make good product requirements out of this? And then I took that and put it into Cursor Composer, told it all of the things. And it actually only was three files. That's all it needed to be. And it was two or three back and force. Within 45 minutes, I had this thing working, which was crazy and I haven't touched any more of that. That is crazy. It's wild. You even have UI in the terminal, which is hard to do. And have them aligned and that stuff. That is incredible. And it did all of that for me. And I actually didn't change a single line of code from the UI or the way that these things were. I messed with the prompts manually because prompts are important. But in terms of how everything worked, I was , no, no, no. From idea to actual app, I'm going to work on that. So yeah, it was about 45 minutes. If I built it, I could have spent more time on it, but I was , you know what? No, this is everything I needed to be right now. That's that's incredible. Great job. Wow. That's very inspiring that you're able to come up with this idea. you said, it's burning your head or something. You just want to get it out and start to exercise that part of your brain. And I definitely want to encourage more engineers who are in this camp you. I feel the more experience you have, the better it is for you. And don't, , think that you can't, it's going to take all this time to code. That's another thing. you just put it out there. That's a really great tip about the whisper thing. So just speaking it out, getting out those instructions as much as possible, handing it to an LLM to generate this project plan basically or these steps to get this accomplished or get these features. And then feeding it back into a nice cursor type of thing. It produces. This is awesome. Wow. Just think about this. , yeah, I built this million dollar app just from, you know, this silly idea. of wanting to watch my posture. Whoa. Well, I could have just spent 45 minutes watching an episode of something on Netflix, or I could just walk in here and blast out a quick app, you know? And that's my favorite part about this podcast, or maybe my second part. It's probably talking to people every week, but my second favorite part of this is building every week. We get to build cool scripts, and they're usually pretty small, but we'll add this to the pit package too, and people can try it if they want to. We were chatting the other day, he's , yo, my posture, 95% the other day. , I was locked in and focused. I'm , okay, yeah, this is more than just demoware. This is actually something he uses, so loved it. Wow. All right, Mike, let's see what you got here. I made a tool called Fuzzy Puffin. But before I show you what Fuzzy Puffin is, I just want to talk about some of the technologies behind it. So there's this thing called LAMA file, and I love LAMophile because it allows you to convert an LLM in GGUF form into an executable that can be passed between different OSES. So if you're running a Mac and your buddy's running on a Windows, you can actually wrap up an LLM in a LOMophile, sends to them, and they can execute it. So they have a list of ones you can download, which is a pretty good list, but what's really cool is you can turn different LLMs into it. So I took my favorite model, which is Hermes 3 from News Research, just the Lava 3.1-8B model, wrapped in the Lama file. So I have my LLM running locally. And then there's this other concept which I came across recently, grammar files, where you can actually have the LLM output a specific format or limited amount of vocabulary. So you can ask you whatever you want, And instead of a long-winded answer with preamble in extra context, it'll just be what you want. So if you want a yes-noe, you can limit it to just that. Or you can limit it to a certain set of scripts. So with our PIP package, the Tully's PIP package, you can see we have nine tools in it now, and we're building it all the time. It's expanding. So it's kind of tougher people to keep track of. So with Fuzzy Puffin, what it is, it's an LLM interface allowing you to use natural language to interact with the package. So you run it, you get Fuzzy Puffin. And then you just get into natural language query. So let's say, I need some help with my upcoming product launch. Send through the voice, and then it processes that, and it goes through the list of scripts, and it realizes that it wants to call AI marketing plan, which is one tie put together. So even though I didn't mention an networking plan or whatnot, because it knew the general concept we were going for, it fired it up. Let's try one other. I really need some help organizing my day. Fuzzy Puffin takes it, looks at all the mapping of the different things. scripts we have available in PIPPackage and knows to use AI prioritize. And I'll just do one last example. I really want to get in touch with the guys from tool use. Fire that through and it'll know to run tool use contact. So they can reach out to us whether you want to suggest a guest, request a topic, give them feedback. This allows you to no longer need to understand all the syntax, all the commands for a PIP package. You can just tell it what you need to do and it will route you to the best script tool I did voice separately. You've think about throwing voice into it since Ty put in shallow Graham in there, but this is just the abstraction that allows you to have a fuzzy function to operate your computer rather than have to be very deterministic with the way you command it. So that's my latest addition to the package. Wow, that's so, so cool. I love that you guys are actively building because that helps you understand all these different types of things. You know, I saw a Lama file at the AI engineering conference earlier this year and i was wow this is really cool and you know you're kind of some of the first use cases i'm seeing of actually making something really useful because that's my problem too is i develop a lot on the mac but my friends have PCs so how how would i give the script to have them run it right it's oh so i this idea that you're using with lama file to kind of spread this around and then this other thing i mean there's so many potential possibilities here of taking this and then having some type of tool use just for your machine right so open interpreter, these other types of things that can control your computer. And so now you're giving an open interpreter system, which means that you can just give us some voice commands to run very specific tasks on your computer that you say, I always do this script or I always do this type of thing on my machine, go do that. So I that workflow. That's really, really cool. Amazing. Thank you. Yeah. One thing with open interpreter that we love is it will dynamically generate the script. So you can go from absolutely nothing to functionality in action. With this, it takes your existing scripts you have. So say, you know, using cursor, put together something, but then it allows you to use whatever natural language you want to route towards there. So it's kind of two different sizes of the same coin, but satisfying a different use case. Wow. That's inspirational. This is super cool. Yeah. I'm honored that it picked my scripts too, Mike. I think that's a good one. This week, right? Yeah. Yeah. We have this little weird competition where every week we're , all right, who won the demo this week. So that's what we play into every week. We really want to make at some point, I guess I'm kind of spilling inner secrets here, Mike, but that's all right. If people are listening, that's fine. But we want to make, , a LLM judge that every week would mark whichever demo is cooler. And there's probably some sort of cool, , hackathon that you could create where everyone makes submissions, but it's all judged by LLMs. It would be super interesting. Take the humans out of it and just let the LLMs pick. That's a really cool idea. Well, that was pretty fun. Is there anything else we want to cover this week? Yeah, I think as far as everything that's going on, I mean, we're almost at the year mark when GPT4 has been launched, right? Isn't that really interesting that a year ago, I feel today, , you know, Sam Altman was being outstead and going to come back to Open AI. You know, GPT4 was just released and kind of changing the world. People thought, , you know, computers were going to take over. The people's processing for coding is, you know, improved. quite a bit. And then, you know, several months following that, you know, just kind of recounting the history here of what just kind of happened this last year. And then what's happened from there to the last three to four months since Claude's sonnet 3.5 came out to improve coding, right? I feel that was that moment three to four months ago that we're all going to be talking about. We're , not only did we have code and we stopped begging the LLMs to, you know, please, please, please, I'll tip you $20 to finish this piece of code to that he just now does the code. I always to try to think more into the future of , you know, what is it the things that, you know, those kids are going to be talking about? Because it feels every year in AI feels seven years in normal lives. And these are the types of things that , you know, we have to think about as far as as these systems continue to improve, you know, we have to tie together all our different experiences to make them even more rich. So the capabilities are there. And then we have to then tie in these types of, you know, techniques. to produce really great app experiences. So I encourage everyone to just go build. If you have some idea in mind, I'm just so impressed with the two of you guys. She's just , I had this idea. I'm going to build it really quick this week. And a couple of voice prompts and, you know, talking to Claude or cursor or whoever your choice of IDE, you can produce code. And to me, code is the magic thing that you can produce that just kind of out of thin air to produce economic value. And typically, have to go plant a seed, wait for that to, you know, water, germinate and everything, and it produces some economic value. You would have to work with your hands and get wood and put it together to make a house and produce economic value. Or today, it's literally your mind and your voice and anything that you can put into a computer, you can have some economic value out. And I definitely encourage people to go build as much as possible. If you have ideas in your head, even if it sounds silly, just go build it. You're going to learn a lot in this process and don't give up. I feel that's really important. Sometimes you'll run into an issue. You're debugging it. It's just part of the process. And this is kind of lifting weights. If you've never lifted weights before and you hit a bug, you're going to run into that, you know, search for ways how you can debug it, reach out to start building communities around you to help, you know, get these types of people. People are really willing to help. People have helped me throughout my lifetime to help me get to where I am today. And I want to help others as well to do the same. And I'm so glad that you guys have this type of platform and this show available to reach out and to build that community, to encourage others to build. And I encourage you to continue to share back what you know with the rest of the world, because internet will live on forever, and this conversation will be part of some type of AI system in the future as well. I think the last year's been crazy. I think, , there's been clear improvements in, , language models and text, obviously, where I think it's the most clear in image models, video, audio, things Suno came out. And that, was just kind of something. And now you hear a song, you can't be 100% sure if it was human generated or not. , it's that good. And video, too. , there's, I have been fooled on YouTube shorts on videos that I'm , I have to watch it back the third or fourth time. And then I realize it might be AI, but we're at that level of quality in both audio, video, images. , it's, it's been such an insane year. And it feels every month there's a new thing. I mean, stable of fusion three to half just came out and the new flux models and, I don't Super crazy. I'm wondering how if this was this LLM wall that were the people to keep talking about and how GBT4. Yeah, I see you shaking your head, Mike. I kind of feel the same way. I mean, maybe there's some amount of wall, but I'm scared for the wall because we're going to blow right past it, I feel . Yeah, yeah. I think there's approaches to things that are the reason kind of why we build is because it takes a village, really. It's not going to be a single company. It's not going to be one person breaking through. It's going to be the world coming together and figuring out new techniques for how we should be doing things. And I think that's what's kind of fun is that we all can play and we find new things. And literally I just, I'm still blown away how , you know, chat GPT is sort of an accident. You know, it's this paper was published many years ago about, you know, attention is all you need. And how that , you know, that single thread of things has now birthed this whole area. that we're using today. And there could be something that's already out there that, you know, we just, it's already been published. And we just have to do that, right? It was , who knows, right? And people are now actively in this discovery phase. So, yeah, that's really cool. And just to echo what you said, Ray, I encourage everyone keep experimenting, building, sharing, and we're just going to come to something really cool. Before we let you go, is there anything you want to share with the audience? Any ways that can keep up with you? Totally. Yeah. I'd definitely go check out Ray Fernando, 1337. You can find me on X. You could also find me on YouTube. Rayferndando.a.i if you want to stay up to date with all my latest blog and postings and things there. I'm going to be launching raytranscribes.com very, very shortly. So by the time this podcast comes out, just go check out Raytranscribes.com. If you are a person creating social media or posting stuff out and you need transcripts done, hit me up because I'll actually be able to get you a 50% discount. I want to encourage people who are in this space to build to make it as fast as possible to get those transcripts. As you can see, how rich your transcripts can turn into blog posts and other things, that's just the first step of many. And this is something that I'm building to give back to folks as well. So yeah, ray transcribes.com. Check out Rayferndo. com. And on socials, Ray Fernando 1337 is probably the best handle you can find. And definitely check out your live streams. I've been a huge fan of them. It's always fun to drop in and lots of knowledge there. Cool. Yeah, that's right. Yes. The live streams, yeah. I do live streaming, interviewing guests just you guys. and building stuff live as well and just kind of really having fun, sharing my passion back. So yeah, thank you guys so much for having me on the show. This is a huge pleasure, and I'm just honored. I'm cool. I'm excited to be one of the guests here before you guys reach a million views. This is going to be really cool to look back and be , yeah, some of the early guests here. Well, thank you so much, right. We appreciate all the kind words, and we'd love to have you back on in the future episode. Cool, but yeah, I think that's about it. Thanks for being on. Awesome. Thanks, guys. You know,