The Local AI Journey: Overcoming the GPU Bottleneck & The Rise of Small Language Models

 

The Local AI Journey: Overcoming the GPU Bottleneck & The Rise of Small Language Models

Context: Tracking my progress with local Large Language Models (LLMs) and autonomous agents, and documenting the hardware hurdles of running AI without a dedicated graphics card.

1. The Current Roadblock: The GPU vs. CPU Divide

Like many others exploring the world of local AI and building custom agents, I’ve hit the inevitable hardware wall. The vast majority of standard LLMs are built to leverage the massive parallel processing power and high memory bandwidth of dedicated graphics cards (VRAM).

Because my current setup relies solely on standard computer architecture without a discrete GPU, trying to run standard models means relying entirely on the system CPU and standard RAM. The result is a massive performance bottleneck, leading to incredibly slow token generation (the speed at which the model types out words) and high system resource drain.

2. The Paradigm Shift: SLMs and Edge Computing

Fortunately, the AI industry is undergoing a massive architectural shift that plays directly to my situation. Instead of just chasing massive, multi-billion parameter models that require server-grade hardware, developers are aggressively focusing on SLMs (Small Language Models) and Edge AI (running models directly on consumer devices).

We are seeing incredible optimization trends driven by the desire to run AI natively on mobile phones and low-power laptops. This optimization trickles down to standard PC hardware in two major ways:

• Advanced Quantization: This is the process of compressing a model's weights (often from 16-bit to 4-bit or lower) so it takes up significantly less memory without losing its intelligence. The GGUF format is currently the gold standard for this, as it is specifically designed to split the workload and run efficiently on CPU and system RAM.

• The Rise of High-Performance 1B to 3B Models: Models in the 1-billion to 3-billion parameter range (like Microsoft's Phi series, Google's Gemma, or Alibaba's Qwen) are becoming incredibly smart. They require a fraction of the memory, making them perfect candidates for non-GPU setups.

3. Looking Ahead (The 6-Month Horizon)

The pace of open-source innovation right now is staggering. What required a high-end graphics card six months ago is now running on a modern smartphone.

Because of this trajectory, I don't necessarily need to rush out and buy an expensive new GPU just yet. Within a matter of weeks or months, the optimization of CPU-bound execution frameworks (like Ollama, llama.cpp, or LM Studio) paired with highly dense, ultra-efficient small models should allow me to run a perfectly capable, highly responsive agent entirely on my current machine.

🛠️ Status Check & Next Steps for My Future Self:

When reviewing this log down the line, check the status of the following:

• Did I stick with CPU execution, or did the requirements change?

• Are GGUF models still the standard for CPU offloading, or has a newer format superseded them?

• How are the current 1B–3B parameter models performing for my autonomous agent tasks compared to the slow performance I'm experiencing today?

Large Language Models (LLMs)

 Like many others right now, I’ve jumped headfirst into exploring local Large Language Models (LLMs) and building my own AI agents. It’s a fascinating space, but I’ve quickly run into a classic hardware bottleneck: my current machine doesn't have a dedicated graphics card (GPU), which makes running these models locally incredibly sluggish.

However, the silver lining is the rapid shift in how these models are being built. There is a massive industry trend toward optimizing smaller, highly efficient LLMs designed to run on low-power devices like mobile phones.

By extension, this means a lot of development is going into making models run efficiently on standard computer CPUs and system RAM, without the absolute necessity of a beefy graphics card. Given the current, staggering pace of AI development, I’m optimistic that a perfectly capable, highly optimized local LLM that runs smoothly on my current hardware setup is only a matter of weeks away.

Note to self: Keep an eye on smaller parameter models (1B–3B) and CPU-optimized formats as this space evolves.

LLM's

Like many in the tech space, I have recently been exploring the capabilities of local Large Language Models (LLMs) and building my own AI agents. However, one of the primary bottlenecks I’ve encountered is hardware limitations; my current setup lacks a dedicated GPU, making local execution quite challenging.

Fortunately, the industry is shifting rapidly. With the growing trend toward optimizing smaller, high-performance models designed to run locally on mobile devices, we are seeing a parallel push for efficient CPU-bound execution. Given the staggering pace of open-source innovation, I am optimistic that running a highly capable, optimized LLM entirely on standard consumer hardware will be seamless in the very near future.

2026

 2026

So its 2026 and its take until today to do a blog post.

I have been very busy writing a lot of music and so have not do this blog or even my other websites.

So this blog is not dead just slightly under used.

No Really Still Here

 No ! really this blog of mine is still alive, its just that I have been spending a lot of mine time on two very different things.

The first one is writing and composing music, and if you look back though this blog you can see that I have been active in the music scene for a long time, but only recently have I manged to get the tools to allow me to use vocals in my music work. I am currently trying to find a balance between my instrumental work and my vocal works. At this time I have yet to find that balance.

My music bank which has both vocal and instrument music tracks/ works has about 1000 items, so I have a lot of music that I have made over the years. If anything its got worse as I seem to be writing several tracks a day sometimes!

The second thing that I have been doing with my time is running.

I currently run Parkrun every Saturday possible, a Parkrun is 5KM

I also run distances up to 30KM and I am in training for the London Marathon in April 2026 which is 42.2KM so I have my work cur out in getting to the distance and also working out my feeding and drinking over that distance.

I want to finish the run and would love to do it in less than 5 hours. These are both hard tasks but I am going to put the effort in and hope that I get the result I need.

Still Here

So I am still here, but have been very busy writing and composing some vocal songs.

I have jumped on the AI bandwagon, and have got Suno.AI singing my lyrics, I must admit that the results are very very good.

Its great to finally hear my lyrics being sung.

 Midnight Bloom

(Verse 1)

Streetlights paint the rain-slicked street,

A hazy glow, a bittersweet

Recall of laughter, whispered low,

Before the silent afterglow.

I trace your ghost on window panes,

The memory of summer rains.

The scent of you, a phantom trace,

A vacant space, an empty space.

(Pre-Chorus)

And every beat my heart takes flight,

Towards a love that burned so bright,

Then faded fast into the night,

Leaving me lost, without your light.

(Chorus)

Midnight bloom, a fragile thing,

The promise whispered on love's wing,

Now a broken, silent string,

An endless ache, my heart takes wing.

Midnight bloom, forever gone,

The echo of a sweet love song,

My heart still beats, though love is wrong,

And in the dark, I still sing on.

(Verse 2)

The cafe where we first met,

Still holds the ghost of your regret.

The words we spoke, I can't forget,

A future painted, then unmet.

I walk these streets, a lonely soul,

Searching for love to make me whole,

But every path, a story told,

Of a love that's growing cold.

(Pre-Chorus)

And every beat my heart takes flight,

Towards a love that burned so bright,

Then faded fast into the night,

Leaving me lost, without your light.

(Chorus)

Midnight bloom, a fragile thing,

The promise whispered on love's wing,

Now a broken, silent string,

An endless ache, my heart takes wing.

Midnight bloom, forever gone,

The echo of a sweet love song,

My heart still beats, though love is wrong,

And in the dark, I still sing on.

(Bridge)

But maybe someday, I'll find my way,

Back to the light of a brighter day,

A love reborn, to light the way,

And banish shadows far away.

(Chorus)

Midnight bloom, a fragile thing,

The promise whispered on love's wing,

Now a broken, silent string,

An endless ache, my heart takes wing.

Midnight bloom, forever gone,

The echo of a sweet love song,

My heart still beats, though love is wrong,

And in the dark, I still sing on.

(Outro)

Oh-oh-oh, midnight bloom,

Oh-oh-oh, fading soon.

Oh-oh-oh, midnight bloom,

Lost in the night's dark gloom.

City of Shadows

(Verse 1)

Westminster Bridge, a silhouette,

Against a bruised and fading sunset.

The Thames, a ribbon, dark and deep,

Whispering secrets it will keep.

A chill wind stirs, a mournful sigh,

As shadows lengthen, and secrets lie.

(Chorus)

City of shadows, where mysteries play,

Beneath the watchful London Eye's cold ray.

From St. Paul's dome to the Shard's sharp gleam,

Lost in the labyrinth, a half-remembered dream.

London Bridge echoes, a silent plea,

Lost in the darkness, the city watches me.

(Verse 2)

The cobbled streets, a maze of night,

Each shadowed alley holds a hidden fright.

The whispers follow, soft and low,

Where secrets bloom and shadows grow.

I walk alone, with hurried pace,

Haunted by time, and by this place.

(Chorus)

City of shadows, where mysteries play,

Beneath the watchful London Eye's cold ray.

From St. Paul's dome to the Shard's sharp gleam,

Lost in the labyrinth, a half-remembered dream.

London Bridge echoes, a silent plea,

Lost in the darkness, the city watches me.

(Bridge)

The river's current pulls me down,

Into the depths of this haunted town.

A fleeting glimpse, a shadowed face,

Lost in this ever-changing space.

(Chorus)

City of shadows, where mysteries play,

Beneath the watchful London Eye's cold ray.

From St. Paul's dome to the Shard's sharp gleam,

Lost in the labyrinth, a half-remembered dream.

London Bridge echoes, a silent plea,

Lost in the darkness, the city watches me.

(Outro)

The city sleeps, but secrets wake,

In this London night, for goodness sake.

The shadows dance, a haunting show,

In the city's heart, where mysteries grow.

 Masquerade of Midnight

**Act I: The Masquerade's Unveiling**

**(Overture – A majestic orchestral prelude, building in intensity.  Soprano voice enters softly.)**

(Verse 1)

The powder dusts the air, a whispered hush descends,

As silk and satin swirl, where pleasure never ends.

My crimson lipstick gleams, a secret in the light,

Reflecting candle flames, burning ever bright.

My emerald eyes observe, a silent, knowing gaze,

Across the crowded room, in this enchanting maze.

**(Chorus – Full orchestral accompaniment, powerful and soaring.)**

Oh, the silken whispers dance, upon the perfumed breeze,

And laughter fills the hall, beneath the rustling trees

of swaying mini-skirts, a kaleidoscope of mirth,

A thousand smiles ignite, to celebrate the earth!

(Verse 2)

Long, raven tresses flow, cascading down my back,

A cascade of dark night, upon this vibrant track.

The rhythm of the strings, a pulse within my soul,

As whispered promises, begin to take their toll.

He watches from afar, his gaze intense and deep,

While secrets intertwine, the mysteries we keep.

**(Chorus – Full orchestral accompaniment, powerful and soaring.)**

Oh, the silken whispers dance, upon the perfumed breeze,

And laughter fills the hall, beneath the rustling trees

of swaying mini-skirts, a kaleidoscope of mirth,

A thousand smiles ignite, to celebrate the earth!

**(Act II: The Dance of Desire)**

(Verse 3)

The music swells and soars, a passionate embrace,

As bodies intertwine, in this enchanted space.

My heart beats like a drum, a frantic, urgent plea,

Lost in the melody, completely wild and free.

The scent of his cologne, a heady, sweet perfume,

Envelops me in dreams, dispelling all the gloom.

**(Aria – A passionate, extended solo, showcasing the soprano's vocal range and emotional depth.)**

(The soprano sings of longing, desire, and the intoxicating power of the moment, employing intricate melismatic runs and dramatic vocalizations.  The orchestra provides a lush and emotive accompaniment.)

**(Act III: Dawn's Embrace)**

(Verse 4)

The first rays of the dawn, paint the sky with gold,

As shadows start to fade, and stories start to unfold.

The night's enchantment ends, a bittersweet farewell,

To moments unforgettable, a spell we knew so well.

But in my memory's keep, the echoes will remain,

Of laughter, whispered secrets, and the joy of love's sweet reign.

**(Chorus – Full orchestral accompaniment, slightly slower tempo, reflecting a sense of peaceful reflection.)**

Oh, the silken whispers fade, upon the morning breeze,

But memories linger on, among the rustling trees,

Of swaying mini-skirts, a kaleidoscope of mirth,

A thousand smiles ignite, to celebrate the earth!

**(Outro – The orchestra fades out slowly, leaving a lingering sense of beauty and wonder.)**

 Midnight Bloom

(Verse 1)

City slicker, heart a-flutter, lipstick smudged, a whispered hush

He caught my eye across the bar, a stolen glance, a nervous blush

My mini skirt, a playful dare, my long hair cascaded down

He saw the fire in my soul, a love that knows no bounds

(Chorus)

Oh, the rhythm of the night, a heartbeat in the street

He pulls me close, beneath the neon glow, our bodies meet

The city hums a soulful tune, a melody of desire

In his arms, I'm blooming late, a midnight, burning fire

(Verse 2)

Silk tights and shadowed rooms, the scent of rain, a gentle breeze

His fingers trace the lines of my dress, a touch that sets me free

He whispers secrets in my ear, words lost in the hazy light

We dance among the fallen stars, bathed in the moon's soft might

(Chorus)

Oh, the rhythm of the night, a heartbeat in the street

He pulls me close, beneath the neon glow, our bodies meet

The city hums a soulful tune, a melody of desire

In his arms, I'm blooming late, a midnight, burning fire

(Bridge)

Time stands still, a moment caught, a love beyond compare

He's the missing piece I've longed for, a love beyond repair

This feeling's deep, a soulful ache, a comfort in his hold

A story etched in stolen glances, a love yet to unfold

(Chorus)

Oh, the rhythm of the night, a heartbeat in the street

He pulls me close, beneath the neon glow, our bodies meet

The city hums a soulful tune, a melody of desire

In his arms, I'm blooming late, a midnight, burning fire

(Outro)

Midnight bloom, a love in bloom, under city lights so bright

Burning slow, a love aglow, a midnight, tender light

 Sunset Boulevard Serenade

(Verse 1)

City lights paint the evening gold,

Another day done, a story told.

He's waiting there, beneath the glow,

My heart beats fast, a restless flow.

Long legs, short skirt, a summer breeze,

Anticipation sets my soul at ease.

(Chorus)

Hand in hand, we stroll along,

The rhythm of our footsteps, a sweet song.

Kisses stolen, beneath the sky,

Stars above, as we walk by.

Long hair, catching the fading light,

Everything feels perfectly right.

(Verse 2)

Date nights whisper in the quiet streets,

Laughter echoes, our love repeats.

His eyes hold mine, a universe untold,

In every touch, a story unfolds.

The scent of summer, warm and sweet,

Our perfect moments, can't be beat.

(Chorus)

Hand in hand, we stroll along,

The rhythm of our footsteps, a sweet song.

Kisses stolen, beneath the sky,

Stars above, as we walk by.

Long hair, catching the fading light,

Everything feels perfectly right.

(Bridge)

Time stands still, as the sunset burns,

A love like ours, forever returns.

In this moment, nothing compares,

Beyond the city, beyond all cares.

(Chorus)

Hand in hand, we stroll along,

The rhythm of our footsteps, a sweet song.

Kisses stolen, beneath the sky,

Stars above, as we walk by.

Long hair, catching the fading light,

Everything feels perfectly right.

(Outro)

Oh, Sunset Boulevard serenade,

A love like ours, will never fade.

Walking hand in hand, into the night,

Bathed in the moon's gentle light.

 AI Singing

So I have just found out that there are some AI systems that might be able to actually sing my lyrics!

This is great news.

The bad news is that it looks like I will have to pay to use any of the better ones out there so I may give it a go if they give demo's that sound anygood.

 Date Knight Fire

Verse 1:

Emerald eyes, a smile so bright,

Across the room, a dazzling light.

He laughs, his words like summer rain,

Washing away all trace of pain.

My heart takes flight, then plummets down,

A silent scream, a stifled frown.

Long, dark hair, a silken stream,

Reflecting candlelight's soft gleam.

Chorus:

A date-knight scene, a stolen glance,

My jealousy, a wicked dance.

His lips curve up, a whispered word,

A secret shared, unheard, absurd,

That I should feel this burning fire,

This desperate, all-consuming pyre.

He's got my dream, a perfect prize,

Reflected in his laughing eyes.

Verse 2:

I watch them leave, a graceful sway,

My bitter thoughts, they steal the day.

The music plays, a mournful tune,

A lonely heart, beneath the moon.

His touch, I crave, a phantom feel,

A wound that time will never heal.

But still I hope, a foolish thing,

That he might notice, my silent sting.

Chorus:

A date-knight scene, a stolen glance,

My jealousy, a wicked dance.

His lips curve up, a whispered word,

A secret shared, unheard, absurd,

That I should feel this burning fire,

This desperate, all-consuming pyre.

He's got my dream, a perfect prize,

Reflected in his laughing eyes.

Bridge:

The city lights, a blurry haze,

Lost in the echo of yesterday's days.

This bitter taste, this aching heart,

Torn apart, right from the start.

I try to hide, this burning shame,

But envy whispers his name.

Chorus:

A date-knight scene, a stolen glance,

My jealousy, a wicked dance.

His lips curve up, a whispered word,

A secret shared, unheard, absurd,

That I should feel this burning fire,

This desperate, all-consuming pyre.

He's got my dream, a perfect prize,

Reflected in his laughing eyes.

Outro:

Emerald eyes… a dazzling light…

Lost in the shadows of the night.

 Watercolour Tears

Verse 1:

Lost in the echoes of a date night gone wrong,

The memory lingers, like a bittersweet song.

Your green eyes, a canvas of a thousand untold tales,

Now just a blur, lost in the autumnal gales.

Chorus:

And the city lights blur, a watercolour of tears,

As I trace the outline of your ghost across the years.

Your smile, a faded photograph, creased and worn,

Leaving me breathless, eternally forlorn.

Verse 2:

Remember that cafe, the steam rising high,

Your long hair cascading, beneath the candlelight sky.

Your lips, a whispered promise, brushed softly against mine,

A memory I hold close, forever entwined.

Chorus:

And the city lights blur, a watercolour of tears,

As I trace the outline of your ghost across the years.

Your smile, a faded photograph, creased and worn,

Leaving me breathless, eternally forlorn.

Bridge:

A single touch, a fleeting glance, a whispered goodbye,

Left me searching for answers, beneath a starless sky.

The laughter, the music, now a ghost in the air,

Echoing softly, a silent despair.

Chorus:

And the city lights blur, a watercolour of tears,

As I trace the outline of your ghost across the years.

Your smile, a faded photograph, creased and worn,

Leaving me breathless, eternally forlorn.

Outro:

Oh, the city lights blur...

Lost in the memory, of your smile...

Forever forlorn...