Unwanted role prompts in streaming

[akashnambiar]

The bug
When streaming the LLM output, prior role prompts are being included in the output. This issue was previously reported in (#943) with version 0.1.16, but it still persists in 0.2.0. There is likely a way to hardcode which outputs are from the assistant, but if there is a different recommended approach to streaming or a fix, please let me know.

To Reproduce

async def stream_output(lm):
    @guidance
    def llm_streamer(lm):
        with assistant():
            lm += gen("answer")
        return lm
        
    with system():
        lm += "You are a helpful assistant"

    with user():
        lm += "Write me a poem in 10 words"

    for token in lm.stream() + llm_streamer():
        yield str(token)
    
async def main():
    try:
        model = models.OpenAI("gpt-4o-mini")
        async for token in stream_output(model):
            print(token)
            print()
    except Exception as e:
        print(f"Unexpected error: {e}")

if __name__ == '__main__':
    asyncio.run(main())

Last Couple of outputs:

<|im_start|>system
You are a helpful assistant<|im_end|>
<|im_start|>user
Write me a poem in 10 words<|im_end|>
<|im_start|>assistant
Whispers of dawn, dreams take flight, hearts entwined, love's

<|im_start|>system
You are a helpful assistant<|im_end|>
<|im_start|>user
Write me a poem in 10 words<|im_end|>
<|im_start|>assistant
Whispers of dawn, dreams take flight, hearts entwined, love's light

<|im_start|>system
You are a helpful assistant<|im_end|>
<|im_start|>user
Write me a poem in 10 words<|im_end|>
<|im_start|>assistant
Whispers of dawn, dreams take flight, hearts entwined, love's light.

System info (please complete the following information):

• OS (e.g. Ubuntu, Windows 11, Mac OS, etc.): Mac OS 14.6.1
• Guidance Version (guidance.__version__): 0.2.0