Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Wandb on ludwigai/ludwig-ray-gpu:latest + ray throws AttributeError: module 'pydantic.fields' has no attribute 'ModelField' #3978

Open
rahulvramesh opened this issue Mar 23, 2024 · 0 comments
Labels
bug Something isn't working dependency llm Large Language Model related ray

Comments

@rahulvramesh
Copy link

Describe the bug
Wandb on ludwigai/ludwig-ray-gpu:latest + ray throws AttributeError: module 'pydantic.fields' has no attribute 'ModelField'

To Reproduce
Steps to reproduce the behavior:
1, Setup Ray Cluster With Docker Image (latest tag)
2, Submit Job

ray job submit \
    --address=http://<ip>:8265 \
    --runtime-env-json='{"conda": {"channels":["conda-forge"],"dependencies": ["wandb"]},"setup_timeout_seconds": 30,"working_dir": ".", "env_vars": {}}' \
    -- ludwig train --config llama-config.yml \
    --dataset "ludwig://alpaca" --wandb

Note : by default wandb triggers no module found issue.

Please provide code, yaml config file and a sample of data in order to entirely reproduce the issue.
Issues that are not reproducible will be ignored.

model_type: llm
base_model: TinyLlama/TinyLlama-1.1B-Chat-v1.0


adapter:
  type: lora

prompt:
  template: |
    ### Instruction:
    {instruction}

    ### Input:
    {input}

    ### Response:

input_features:
  - name: prompt
    type: text
    preprocessing:
      max_sequence_length: 256

output_features:
  - name: output
    type: text
    preprocessing:
      max_sequence_length: 256

trainer:
  type: finetune
  learning_rate: 0.0001
  batch_size: 1
  gradient_accumulation_steps: 16
  epochs: 1
  learning_rate_scheduler:
    warmup_fraction: 0.01

backend:
  type: ray
  cache_dir: s3://fine-tuning-testing/cache
  trainer:
    use_gpu: true
    num_workers: 2
    strategy:
      type: deepspeed
      zero_optimization:
        stage: 3
        offload_optimizer:
          device: cpu
          pin_memory: true
      bf16:
        enabled: true

Expected behavior
should install wandb and start pushing metrics

Error

tatus message: Unexpected error occurred: The actor died because of an error raised in its creation task, ray::_ray_internal_job_actor_raysubmit_LFQdYns3Zsi5495P:JobSupervisor.__init__() (pid=53445, ip=172.31.77.187, repr=<ray.dashboard.modules.job.job_manager.JobSupervisor object at 0x7faa547b5f70>)
  File "/tmp/ray/session_2024-03-23_07-37-24_381042_194/runtime_resources/conda/66324431b46322e8e8f74b368def6d448f095eb8/lib/python3.8/concurrent/futures/_base.py", line 437, in result
    return self.__get_result()
  File "/tmp/ray/session_2024-03-23_07-37-24_381042_194/runtime_resources/conda/66324431b46322e8e8f74b368def6d448f095eb8/lib/python3.8/concurrent/futures/_base.py", line 389, in __get_result
    raise self._exception
  File "/tmp/ray/session_2024-03-23_07-37-24_381042_194/runtime_resources/conda/66324431b46322e8e8f74b368def6d448f095eb8/lib/python3.8/site-packages/ray/util/serialization_addons.py", line 58, in apply
    register_pydantic_serializer(serialization_context)
  File "/tmp/ray/session_2024-03-23_07-37-24_381042_194/runtime_resources/conda/66324431b46322e8e8f74b368def6d448f095eb8/lib/python3.8/site-packages/ray/util/serialization_addons.py", line 21, in register_pydantic_serializer
    pydantic.fields.ModelField,
AttributeError: module 'pydantic.fields' has no attribute 'ModelField'

Environment (please complete the following information):

  • OS: Ubuntu
  • Version 20
  • Python 3.8.13

Additional context

  • tried by downgrading pydantic, which triggers more module errors.
@alexsherstinsky alexsherstinsky added the llm Large Language Model related label Jul 26, 2024
@mhabedank mhabedank assigned mhabedank and unassigned mhabedank Oct 21, 2024
@mhabedank mhabedank added bug Something isn't working ray dependency labels Oct 21, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working dependency llm Large Language Model related ray
Projects
None yet
Development

No branches or pull requests

3 participants