Skip to content

[improve] More pythonic way for parameter local_params in class task #53

Merged
jieguangzhou merged 7 commits into
apache:mainfrom
jieguangzhou:task-local-parameters
Dec 20, 2022
Merged

[improve] More pythonic way for parameter local_params in class task #53
jieguangzhou merged 7 commits into
apache:mainfrom
jieguangzhou:task-local-parameters

Conversation

@jieguangzhou

@jieguangzhou jieguangzhou commented Dec 16, 2022

Copy link
Copy Markdown
Member

Brief Summary of The Change

close: #21

# [start workflow_declare]
 r"""
 A tutorial example set local parameter in pydolphinscheduler.
 Method 1:
     task = Shell(..., input_params={"input":"a"}, output_params={"output": "b"})
 Method 2:
     task = Shell(...)
     task.add_in("input", "a")
     task.add_out("output", "b")
 """

 from pydolphinscheduler.core.parameter import ParameterType
 from pydolphinscheduler.core.workflow import Workflow
 from pydolphinscheduler.tasks.shell import Shell

 with Workflow(name="local_parameter_example", release_state="offline") as workflow:

     # Add in task __init__ func
     task_1 = Shell(
         name="task_1",
         command="echo hello pydolphinscheduler",
         input_params={
             "value_VARCHAR": "abc",
             "value_INTEGER": 123,
             "value_FLOAT": 0.1,
             "value_BOOLEAN": True,
         },
         output_params={
             "value_EMPTY": None,
         },
     )

     # Add in task instance
     task_2 = Shell(name="task_2", command="echo hello pydolphinscheduler")

     task_2.add_in("value_VARCHAR", "abc")
     task_2.add_in("value_INTEGER", 123)
     task_2.add_in("value_FLOAT", 0.1)
     task_2.add_in("value_BOOLEAN", True)
     task_2.add_out("value_EMPTY")

     # Task 1 is the same as task 2

     # Other parameter types corresponding to DolphinScheduler
     task_3 = Shell(
         name="task_3",
         command="echo hello pydolphinscheduler",
         input_params={
             "value_LONG": ParameterType.LONG("1000000"),
             "value_DATE": ParameterType.DATE("2022-01-02"),
             "value_TIME": ParameterType.TIME("2022-01-01"),
             "value_TIMESTAMP": ParameterType.TIMESTAMP(123123124125),
             "value_LIST": ParameterType.LIST("123123"),
         },
         output_params={
             "output_INTEGER": ParameterType.INTEGER(100),
             "output_LIST": ParameterType.LIST(),
         },
     )

     workflow.submit()
 # [end workflow_declare]

image

Pull Request checklist

I confirm that the following checklist has been completed.

  • Add/Change test cases for the changes.
  • Add/Change the related documentation, should also change docs/source/config.rst when you change file default_config.yaml.
  • (Optional) Add your change to UPDATING.md when it is an incompatible change.

@zhongjiajie zhongjiajie left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please remember change the example doc in the end

Comment thread src/pydolphinscheduler/examples/parameter_example.py Outdated
@jieguangzhou jieguangzhou added the enhancement New feature or request label Dec 19, 2022
Comment thread docs/source/tutorial.rst Outdated
Comment thread src/pydolphinscheduler/core/task.py Outdated
Comment thread src/pydolphinscheduler/examples/local_parameter_example.py Outdated
Comment thread src/pydolphinscheduler/examples/local_parameter_example.py Outdated
Comment thread src/pydolphinscheduler/examples/local_parameter_example.py Outdated
Comment thread tests/core/test_task.py
Comment thread src/pydolphinscheduler/core/task.py

@zhongjiajie zhongjiajie left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Others LGTM

Comment thread src/pydolphinscheduler/core/task.py Outdated
Comment thread src/pydolphinscheduler/core/task.py
Comment thread src/pydolphinscheduler/core/task.py Outdated
Comment thread src/pydolphinscheduler/core/yaml_workflow.py Outdated
Comment thread docs/source/concept.rst Outdated
@jieguangzhou jieguangzhou self-assigned this Dec 20, 2022

@zhongjiajie zhongjiajie left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Comment thread .gitignore
@jieguangzhou jieguangzhou merged commit 68d2670 into apache:main Dec 20, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[improve] More pythonic way for parameter local_params in class task

2 participants