How I Learn with AI (and used it to Build an MVP Vision Pro App in a Week with no Prior Experience in VisionOS)

I created my MVP Vision Pro app in just 4 days.

This is with no prior experience with the VisionOS SDK. But how is that possible?

Well, I paid a developer to do it…

Just kidding—I was able to do this because I know how to use AI to my advantage. I have developed a mindset of learning with AI as my guide. I’m going to show you how I went about building this with self-guided learning from ChatGPT and will explain why this is a microcosm of where the world is going.

A world where anyone can learn and build anything. We have the tools, now all we need is the will and a way to get the most out of these tools.

First, a brief overview of how to think about self-learning with AI —then I’ll show you my experience from this last week where you’ll get to see my AI self-learning framework was developed over countless hours of practice and iteration.

Let’s dive in.

Learning to Learn: An AI State of Mind

AI is a powerful tool for those who have learned how to learn.

There is a segment of people who will say things like:

— “I don’t even know where to start.”

— “It’s too hard!”

— “What can AI teach us anyway?”

These are dogmas and defense mechanisms for the close-minded.

It is often because we have been programmed by society to be really good at following directions, but never solving open-ended problems.

The first hurdle is knowing the right questions to ask. Luckily, LLMs are very capable of helping us get the dialogue started. When in doubt, I ask what questions I should be asking—most people are deterred by new projects because they don’t even know the rules of the game they’re playing.

This is the first step to learning to learn: asking questions.

My 4-Day Sprint to Building a Vision Pro App

I had no prior knowledge of VisionOS development before starting this project (yes, I am a dev so I have some advantage).

The point is, that I used the knowledge I did have to push the boundaries of what I was capable of. I used ChatGPT as my guide into new territory.

Asking the Right Questions

When learning something new, I ask ChatGPT what questions I should be asking before doing anything else—if you don’t know the rules of the game, you can’t play it very well.

Here’s me asking a Custom GPT for VisionOS the main concepts of the VisionOS ecosystem:

This sets the context form and gives me a big-picture understanding of the ecosystem. I now learned what was even possible and what things to consider before thinking of an idea for an app.

We often get overwhelmed with new territory because we aren’t aware of the things we should be aware of. Using ChatGPT to guide you through this exploratory phase rapidly increases the rate of comprehension—we now know the rules of the game we are playing.

Prompting for Lightning in a Bottle

The next phase is coming up with an idea.

A common hurdle in brainstorming is coming up with the ideas themselves. Having a sounding board for ideation is a great way to get the thought juices flowing:

And just like that, we’ve got some ideas to play with!

A great way to prompt ChatGPT for things that have more complexity is to list out what you want ChatGPT in steps—be specific.

Time to Start Building

The children’s story idea caught my idea—I immediately had a rough idea of how I could do this.

Here’s the MVP idea I mapped out for the short story app:

• Prompts the user to select a base theme for the type of story they want to see.

•  Ask the user to fill out some information on the main character of the story.

• With this information, we can run it through a series of calls to an LLM to generate a short story script and accompanying images for segments of the story.

• We then send all this back to the client and let the user enjoy their custom story.

Pair Programming With AI

With AI as the driver of output, I take on more of an engineering manager role.

The shift in the human role with AI emphasizes the value of meta-cognition—to think at a higher level, see the big picture, and translate that into actionable steps for AI to take.

Do not just copy and paste code!

I use it as a way to get started with new frameworks or engineering challenges that I run into. It’s helpful as a way to talk through my ideas too—a sounding board that is always there, ready to give feedback and ideas.

Challenges Faced

The key challenge I faced was the setup that my original setup called the backend—expecting the story with all accompanying images to be generated and sent back to the client.

But, the time it takes for this MVP pipeline took roughly 6 minutes in total which came with the following issues:

• Client connecting would timeout after 1 minute.

• Waiting for a 6-minute long process is not an optimal experience.

After discussing several workarounds with ChatGPT, we landed on the following implementation:

1. Call the backend to create a new task for generating a story in our Firestore.

2. Use a Firebase cloud function that would be triggered every time a new document is added to the “generate_story_tasks” collection.

This allowed us to have our pipeline work in the background without waiting for a direct response on the client side.

The only downside currently is the user will need to manually refresh their stories page to see if the story is done being generated.

But as the adage goes: “Make it work, make it right, make it fast”—the goal here was to get an MVP version of the app done, which we did.

The MVP

Here is what I was able to get after 4ish days of just coding with ChatGPT by my side and a helpful resource for knowledge. YouTube video demo below:

Future Improvements

I am excited about the concept of this app. I think the Vision Pros offer a whole new dimension to world-building.

I’m happy with where I got in a short amount of time because it gets the basic idea out of my head, and now I can start refining it.

Some improvements I want to make to this include but are not limited to:

1. Better UI/UX overall—pretty basic UI right now (gets the job done but I want it to feel premium).

2. Immersive worlds—I’m starting to explore Apple’s RealityKit more and hope to incorporate an immersive element as I believe it would add more to the story experience.

Insights Gained from the Development Journey

Throughout this journey, several key insights emerged, shedding light on both the potential and the limitations of AI in app development:

1. AI as a Partner in Innovation: AI's role extended beyond a mere provider of information to an active participant in problem-solving. Its capacity to suggest viable solutions underlines the evolving nature of AI tools as partners in innovation.

2. The Importance of a Human Touch: Despite AI's capabilities, the human element—our creativity, intuition, and adaptability—remains indispensable. The final decision-making, especially in addressing unexpected challenges, relied on human ingenuity.

3. Learning to Learn with AI: This experience reinforced the value of learning to learn. AI can offer a vast array of knowledge and suggestions, but the ability to ask the right questions, interpret AI's guidance, and apply it effectively, hinges on our learning agility.

A Future Where We Can Create Anything

My journey of building this is a microcosm of a larger story—AI is breaking the limits of what we once thought was possible.

Those doubts in your head? They are limiting beliefs. The reality is it has never been easier to learn new things and build value.

As we stand on this frontier, it's crucial to remember that our progress is not solely dictated by technology. It's the distinctly human attributes we bring to the partnership with AI that will define our trajectory.

Here's what we need to lay on the table:

• Curiosity: The spark that ignites the quest for knowledge,

• Passion: The fuel that sustains our journey.

• Resilience: To bounce back stronger.

• Vision: A clear picture of what we aim to achieve.

• Creativity: Connecting disparate ideas, to come up with novel ones.

If you can bring these human elements to the table, you’re unstoppable.

I encourage you to try this for yourself next time you can. Pick something that requires you to learn something new, build something new, and be out of your comfort zone. Then, use AI to empower your ability to bring it to fruition.

Looking Forward: The Future of AI-Assisted Creativity

As we stand on the brink of a new era of technological innovation, the possibilities seem boundless.

This project is just one example of how AI can catalyze creativity, allowing individuals to transcend traditional limitations and bring their unique visions to life.

However, the journey also highlights the essential balance between leveraging AI's potential and cultivating our innate human capacities.